Original: D EEP D ENOISING : R ATE -O PTIMAL R ECOVERY OF S TRUCTURED S IGNALS WITH A D EEP P RIOR
Revised: D EEP D ENOISING : R ATE -O PTIMAL R ECOVERY OF S TRUCTURED S IGNALS WITH A D EEP P RIOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks provide state-of-the-art performance for image denoising, where the goal is to recover a near noise-free image from a noisy image.
Revised: Deep neural networks provide state-of-the-art performance for image denoising, where the goal is to recover a near noise-free image from a noisy image.

Original: The underlying principle is that neural networks trained on large datasets have empirically been shown to be able to generate natural images well from a lowdimensional latent representation of the image.
Revised: The underlying principle is that neural networks trained on large datasets have empirically been shown to be able to generate natural images well from a lowdimensional latent representation of the image.

Original: Given such a generator network, or prior, a noisy image can be denoised by ﬁnding the closest image in the range of the prior.
Revised: Given such a generator network, or prior, a noisy image can be denoised by ﬁnding the closest image in the range of the prior.

Original: However, there is little theory to justify this success, let alone to predict the denoising performance as a function of the networks parameters.
Revised: However, there is little theory to justify this success, let alone to predict the denoising performance as a function of the networks parameters.

Original: In this paper we consider the problem of denoising an image from additive Gaussian noise, assuming the image is well described by a deep neural network with ReLu activations functions, mapping a k -dimensional latent space to an n -dimensional image.
Revised: In this paper we consider the problem of denoising an image from additive Gaussian noise, assuming the image is well described by a deep neural network with ReLu activations functions, mapping a k -dimensional latent space to an n -dimensional image.

Original: We state and analyze a simple gradient-descent-like iterative algorithm that minimizes a non-convex loss function, and provably removes a fraction of p 

Revised: We state and analyze a simple gradient-descent-like iterative algorithm that minimizes a non-convex loss function, and provably removes a fraction of p

Original: ´ O p k { n qq of the noise energy.
Revised: ´ O p k { n qq of the noise energy.

Original: We also demonstrate in numerical experiments that this denoising performance is, indeed, achieved by generative priors learned from data.
Revised: We also demonstrate in numerical experiments that this denoising performance is, indeed, achieved by generative priors learned from data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the image or signal denoising problem, where the goal is to remove noise from an unknown image or signal.
Revised: We consider the image or signal denoising problem, where the goal is to remove noise from an unknown image or signal.

Original: In more detail, our goal is to obtain an estimate of an image or signal y ˚ P R n from
Revised: In more detail, our goal is to obtain an estimate of an image or signal y ˚ P R n from

Original: [Equation]
Revised: [Equation]

Original: where η is unknown noise, often modeled as a zero-mean white Gaussian random variable with covariance matrix σ 2 { nI .
Revised: where η is unknown noise, often modeled as a zero-mean white Gaussian random variable with covariance matrix σ 2 { nI .

Original: Image denoising relies on modeling or prior assumptions on the image y ˚ .
Revised: Image denoising relies on modeling or prior assumptions on the image y ˚ .

Original: For example, suppose that the image y ˚ lies in a k -dimensional subspace of R n denoted by Y .
Revised: For example, suppose that the image y ˚ lies in a k -dimensional subspace of R n denoted by Y .

Original: Then we can estimate the original image by ﬁnding the closest point in (cid:96) 2 -distance to the noisy observation y on the subspace
Revised: Then we can estimate the original image by ﬁnding the closest point in (cid:96) 2 -distance to the noisy observation y on the subspace

Original: Y .
Revised: Y .

Original: The corresponding estimate, denoted by ˆ y , obeys
Revised: The corresponding estimate, denoted by ˆ y , obeys

Original: [Equation]
Revised: [Equation]

Original: with high probability (throughout, }¨} denotes the (cid:96) 2 -norm).
Revised: with high probability (throughout, }¨} denotes the (cid:96) 2 -norm).

Original: Thus, the noise energy is reduced by a factor of k { n over the trivial estimate ˆ y “ y which does not use any prior knowledge of the signal.
Revised: Thus, the noise energy is reduced by a factor of k { n over the trivial estimate ˆ y “ y which does not use any prior knowledge of the signal.

Original: The denoising rate (1) shows that the more concise the image prior or image representation (i.e., the smaller k ), the more noise can be removed.
Revised: The denoising rate (1) shows that the more concise the image prior or image representation (i.e., the smaller k ), the more noise can be removed.

Original: If on the other hand the prior (the subspace, in this example) does not include the original image y ˚ , then the error bound (1) increases as we would remove a signiﬁcant part of the signal along with noise when projecting onto the range of the signal prior.
Revised: If on the other hand the prior (the subspace, in this example) does not include the original image y ˚ , then the error bound (1) increases as we would remove a signiﬁcant part of the signal along with noise when projecting onto the range of the signal prior.

Original: Thus a concise and accurate prior is crucial for denoising.
Revised: Thus a concise and accurate prior is crucial for denoising.

Original: Real world signals rarely lie in a priori known subspaces, and the last few decades of image denoising research have developed sophisticated and accurate image models or priors and algorithms.
Revised: Real world signals rarely lie in a priori known subspaces, and the last few decades of image denoising research have developed sophisticated and accurate image models or priors and algorithms.

Original: Examples include models based on sparse representations in overcomplete dictionaries such as wavelets (Donoho, 1995) and
Revised: Examples include models based on sparse representations in overcomplete dictionaries such as wavelets (Donoho, 1995) and

Original: curvelets (Starck et al., 2002), and algorithms based on exploiting self-similarity within images (Dabov et al., 2007).
Revised: curvelets (Starck et al., 2002), and algorithms based on exploiting self-similarity within images (Dabov et al., 2007).

Original: A prominent example of the former class of algorithms is the BM3D (Dabov et al., 2007) algorithm, which achieves state-of-the-art performance for certain denoising problems.
Revised: A prominent example of the former class of algorithms is the BM3D (Dabov et al., 2007) algorithm, which achieves state-of-the-art performance for certain denoising problems.

Original: However, the nuances of real world images are difﬁcult to describe with handcrafted models.
Revised: However, the nuances of real world images are difﬁcult to describe with handcrafted models.

Original: Thus, starting with the paper (Elad & Aharon, 2006) that proposes to learn sparse representation based on training data, it has become common to learn concise representation for denoising (and other inverse problems) from a set of training images.
Revised: Thus, starting with the paper (Elad & Aharon, 2006) that proposes to learn sparse representation based on training data, it has become common to learn concise representation for denoising (and other inverse problems) from a set of training images.

Original: In 2012, Burger et al.
Revised: In 2012, Burger et al.

Original: (Burger et al., 2012) applied deep networks to the denoising problem, by training a deep network on a large set of images.
Revised: (Burger et al., 2012) applied deep networks to the denoising problem, by training a deep network on a large set of images.

Original: Since then, deep learning based denoisers (Zhang et al., 2017) have set the standard for denoising.
Revised: Since then, deep learning based denoisers (Zhang et al., 2017) have set the standard for denoising.

Original: The success of deep network priors can be attributed to their ability to efﬁciently represent and learn realistic image priors, for example via autodecoders (Hinton & Salakhutdinov, 2006) and generative adversarial models (Goodfellow et al., 2014).
Revised: The success of deep network priors can be attributed to their ability to efﬁciently represent and learn realistic image priors, for example via autodecoders (Hinton & Salakhutdinov, 2006) and generative adversarial models (Goodfellow et al., 2014).

Original: Over the last few years, the quality of deep priors has signiﬁcantly improved (Karras et al., 2017; Ulyanov et al., 2017).
Revised: Over the last few years, the quality of deep priors has signiﬁcantly improved (Karras et al., 2017; Ulyanov et al., 2017).

Original: As this ﬁeld matures, priors will be developed with even smaller latent code dimensionality and more accurate approximation of natural signal manifolds.
Revised: As this ﬁeld matures, priors will be developed with even smaller latent code dimensionality and more accurate approximation of natural signal manifolds.

Original: Consequently, the representation error from deep priors will decrease, and thereby enable even more powerful denoisers.
Revised: Consequently, the representation error from deep priors will decrease, and thereby enable even more powerful denoisers.

Original: While deep networks can model complex priors through many parameters and non-linearities, those non-linearities also make their analysis inherently difﬁcult, increasing the gap between theory and practice.
Revised: As the inﬂuence of deep networks in inverse problems grows, it becomes increasingly important to understand their performance at a theoretical level. Given that most optimization approaches for deep learning are ﬁrst order gradient methods, a justiﬁcation is needed for why they do not get stuck in local minima. The closest theoretical work to this question is Bora et al.

Original: As a consequence, there is  a need for theory explaining the success of deep network based priors.
Revised: (2017), which solves a noisy compressive sensing problem with generative priors by minimizing empirical risk. Under the assumption that the network is Lipschitz, they show that if the global optimizer can be found, which is in principle NP-hard, then a signal estimate is recovered to within the noise level. While the Lipschitzness assumption is quite mild, the resulting theory does not provide justiﬁcation for why global optimality can be reached.

Original: Contributions: The goal of this paper is to analytically quantify the denoising performance of deep-prior based denoisers.
Revised: The most related work that establishes theoretical reasons for why gradient methods would not get stuck in local minima, when using deep generative priors for solving inverse problems, is Hand & Voroninski (2018). In it, the authors establish global favorability for optimization of the noiseless empirical risk function. Speciﬁcally, they show existence of a descent direction outside a ball around the global optimizer and a negative multiple of it in the latent space of the generative model.

Revised: This work does not provide a speciﬁc algorithm which provably estimates the global minimizer, nor does it provide an analysis of the robustness of the problem with respect to noise.

Revised: In this paper, we propose the ﬁrst algorithm for solving denoising with deep generative priors that provably ﬁnds an approximation of the underlying image.

Revised: Following the lead of Hand & Voroninski (2018), we assume an expansive Gaussian model for the deep generative network in order to establish this result.

Revised: Contributions: The goal of this paper is to analytically quantify the denoising performance of deep-prior based denoisers.

Original: Speciﬁcally, we characterize thedenoising performance of a simple and efﬁcient algorithm for denoising based on a d -layer generative neural network G : R k Ñ R n , with k ă n , and random weights.
Revised: Speciﬁcally, we characterize the performance of a simple and efﬁcient algorithm for denoising based on a d -layer generative neural network G : R k Ñ R n , with k ă n , and random weights.

Original: In more detail, we propose a gradient method with a tweak that attempts to minimize the least-squares loss f p x q “ 12 } G p x q ´ y }  between the noisy image y and an image in the range of the prior, G p x q .
Revised: In more detail, we propose a gradient method with a tweak that attempts to minimize the least-squares loss f p x q “ 12 } G p x q ´ y } 2 between the noisy image y and an image in the range of the prior, G p x q .

Original: Albeit f is non-convex, we show that the gradient methodwith a tweak yields an estimate ˆ x obeying
Revised: While f is non-convex, we show that the gradient method yields an estimate [Equation] x obeying

Original: [Equation]
Revised: [Equation]

Original: with high probability, where the notation À absorbs a constant factor depending on the number of layers of the network, and its expansitivity,  discussed in more detail later.
Revised: with high probability, where the notation À absorbs a constant factor depending on the number of layers of the network, and its expansitivity, as discussed in more detail later.

Original: Our result shows that the denoising rate of a deep prior based denoiser is determined by the dimension of the latent representation.
Revised: Our result shows that the denoising rate of a deep prior based denoiser is determined by the dimension of the latent representation.

Original: We also show in numerical experiments, that this rate—shown to be analytically achieved for random priors—is also experimentally achieved for priors learned from real imaging data. 
Revised: We also show in numerical experiments, that this rate—shown to be analytically achieved for random priors—is also experimentally achieved for priors learned from real imaging data. [Figure]

Original:  P ROBLEM FORMULATION
Revised:  P ROBLEM FORMULATION

Original: We consider the problem of estimating a vector y ˚ P R n from a noisy observation y “ y ˚ ` η .
Revised: We consider the problem of estimating a vector y ˚ P R n from a noisy observation y “ y ˚ ` η .

Original: We assume that the vector y ˚ belongs to the range of a d -layer generative neural network G : R k Ñ R n , with k ă n .
Revised: We assume that the vector y ˚ belongs to the range of a d -layer generative neural network G : R k Ñ R n , with k ă n .

Original: That is, y ˚ “ G p x ˚ q for some x ˚ P R k .
Revised: That is, y ˚ “ G p x ˚ q for some x ˚ P R k .

Original: We consider a generative network of the form
Revised: We consider a generative network of the form

Original: G

Original: [Equation]

Original: relu

Original: [Equation]
Revised: [Equation]

Original: relu p

Original: [Equation] [Equation] [Equation] , where relu p x q “ max p x, 0 q applies entrywise, W i P R n i ˆ n i ´ 1 , are the weights in the i -th layer, n i is the number of neurons in the i th layer, and the network is expansive in the sense
Revised: where relu p x q “ max p x, 0 q applies entrywise, W i P R n i ˆ n i ´ 1 , are the weights in the i -th layer, n i is the number of neurons in the i th layer, and the network is expansive in the sense

Original: that k “ n 0 ă n 1 ă ¨ ¨ ¨ ă n d “ n .
Revised: that k “ n 0 ă n 1 ă ¨ ¨ ¨ ă n d “ n .

Original: The problem at hand is: Given the weights of the network W 1 . . .
Revised: The problem at hand is: Given the weights of the network W 1 . . .

Original: W d and a noisy observation y , obtain an estimate ˆ y of the original image y ˚ such that }
Revised: W d and a noisy observation y , obtain an estimate ˆ y of the original image y ˚ such that }

Original: ˆ y ´ y ˚ } is small and ˆ y is in the range of G .
Revised: ˆ y ´ y ˚ } is small and ˆ y is in the range of G .

Original:  D ENOISING VIA EMPIRICAL RISK MINIMIZATION
Revised:  D ENOISING VIA EMPIRICAL RISK MINIMIZATION

Original: As a way to solve the above problem, we ﬁrst obtain an estimate of x ˚ , denoted by ˆ x , and then estimate y ˚ as G p ˆ x q .
Revised: As a way to solve the above problem, we ﬁrst obtain an estimate of x ˚ , denoted by ˆ x , and then estimate y ˚ as G p ˆ x q .

Original: In order to estimate x ˚ , we minimize the empirical risk objective
Revised: In order to estimate x ˚ , we minimize the empirical risk objective

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: Since this objective is nonconvex, there is no a priori guarantee of efﬁciently ﬁnding the global minimum.
Revised: Since this objective is nonconvex, there is no a priori guarantee of efﬁciently ﬁnding the global minimum.

Original: Approaches such as gradient methods could in principle get stuck in local minima, instead of ﬁnding a global minimizer that is close to x ˚ .
Revised: Approaches such as gradient methods could in principle get stuck in local minima, instead of ﬁnding a global minimizer that is close to x ˚ .

Original: However, as we show in this paper, under appropriate conditions, a gradient method with a tweak— introduced next—ﬁnds a point that is very close to the original latent parameter x ˚ , with the distance to the parameter x ˚ controlled by the noise.
Revised: However, as we show in this paper, under appropriate conditions, a gradient method with a tweak— introduced next—ﬁnds a point that is very close to the original latent parameter x ˚ , with the distance to the parameter x ˚ controlled by the noise.

Original: In order to state the algorithm, we ﬁrst introduce a useful quantity.
Revised: In order to state the algorithm, we ﬁrst introduce a useful quantity.

Original: For analyzing which rows of a matrix W are active when computing relu p
Revised: For analyzing which rows of a matrix W are active when computing relu p

Original: Wx q , we let
Revised: Wx q , we let

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed weight matrix W , the matrix W ` ,x zeros out the rows of W that do not have a positive dot product with x .
Revised: For a ﬁxed weight matrix W , the matrix W ` ,x zeros out the rows of W that do not have a positive dot product with x .

Original: Alternatively put, W ` ,x contains weights from only the neurons that are active for the input x .
Revised: Alternatively put, W ` ,x contains weights from only the neurons that are active for the input x .

Original: We also deﬁne W 1 , ` ,x “ p W 1 q ` ,x “ diag p W 1 x ą 0 q W 1 and
Revised: We also deﬁne W 1 , ` ,x “ p W 1 q ` ,x “ diag p W 1 x ą 0 q W 1 and

Original: W i, ` ,x “ diag p W i W i ´ 1 , ` ,x ¨ ¨ ¨ W 2 , ` ,x W 1 , ` ,x x ą 0 q W i .
Revised: W i, ` ,x “ diag p W i W i ´ 1 , ` ,x ¨ ¨ ¨ W 2 , ` ,x W 1 , ` ,x x ą 0 q W i .

Original: The matrix W i, ` ,x consists only of the weights of the neurons in the i th layer that are active if the input to the ﬁrst layer is x .
Revised: The matrix W i, ` ,x consists only of the weights of the neurons in the i th layer that are active if the input to the ﬁrst layer is x .

Original: We are now ready to state our algorithm: a gradient method with a tweak informed by the loss surface of the function to be minimized.
Revised: We are now ready to state our algorithm: a gradient method with a tweak informed by the loss surface of the function to be minimized.

Original: Given a noisy observation y , the algorithm starts with an arbitrary initial point x 0 ‰ 0 .
Revised: Given a noisy observation y , the algorithm starts with an arbitrary initial point x 0 ‰ 0 .

Original: At each iteration i “ 0 , 1 , . . .
Revised: At each iteration i “ 0 , 1 , . . .

Original: , the algorithm computes the step direction
Revised: , the algorithm computes the step direction

Original: [Equation]
Revised: [Equation]

Original: which is equal to the gradient of f if f is differentiable at x i .
Revised: which is equal to the gradient of f if f is differentiable at x i .

Original: It then takes a small step opposite to ˜ v x i .
Revised: It then takes a small step opposite to ˜ v x i .

Original: The tweak is that before each iteration, the algorithm checks whether f p´ x i q is smaller than f p x i q , and if so, negates the sign of the current iterate x i .
Revised: The tweak is that before each iteration, the algorithm checks whether f p´ x i q is smaller than f p x i q , and if so, negates the sign of the current iterate x i .

Original: This tweak is informed by the loss surface.
Revised: This tweak is informed by the loss surface.

Original: To understand this step, it is instructive to examine the loss surface for the noiseless case in Figure 1.
Revised: To understand this step, it is instructive to examine the loss surface for the noiseless case in Figure 1.

Original: It can be seen that while the loss function has a global minimum at x ˚ , it is relatively ﬂat close to ´ x ˚ .
Revised: It can be seen that while the loss function has a global minimum at x ˚ , it is relatively ﬂat close to ´ x ˚ .

Original: In expectation, there is a critical point that is a negative multiple of x ˚ with the property that the curvature in the ˘ x ˚ direction is positive, and the curvature in the orthogonal directions is zero.
Revised: In expectation, there is a critical point that is a negative multiple of x ˚ with the property that the curvature in the ˘ x ˚ direction is positive, and the curvature in the orthogonal directions is zero.

Original: Further, around approximately ´ x ˚ , the loss function is larger than around the optimum x ˚ .
Revised: Further, around approximately ´ x ˚ , the loss function is larger than around the optimum x ˚ .

Original: As a simple gradient descent method (without the tweak) could potentially get stuck in this region, the negation check provides a way to avoid converging to this region.
Revised: As a simple gradient descent method (without the tweak) could potentially get stuck in this region, the negation check provides a way to avoid converging to this region.

Original: Our algorithm is formally summarized as Algorithm 1 below.
Revised: Our algorithm is formally summarized as Algorithm 1 below.

Original: Other variations of the tweak are also possible.

Original: For example, the negation check in Step 2 could be performed after a convergence criterion is satisﬁed, and if a lower objective is achieved by negating the latent code, then the gradient descent can be continued again until a convergence criterion is again satisﬁed.

Original: Algorithm

Original: Require: Weights of the network W i , noisy observation y , and step size α ą1: Choose an arbitrary initial point x 0 P R k zt 0 u 2: for i “ 0 , 1 , . . .
Revised: Require: Weights of the network W i , noisy observation y , and step size α ą1: Choose an arbitrary initial point x 0 P R k zt 0 u 2: for i “ 0 , 1 , . . .

Original: do 3: if f p´x i q ă f p 
Revised: do 3: if f p´ q ă f p x q then 4: x

Original:  x
Revised: Ð ´ x

Original: i q then 4: x i Ð ´ x i ; 5: end if 6: 
Revised: ; 5: end if 6: Compute ˜ v [Equation]

Original: Compute ˜ v x i “ p Π 1 i “ d W i, ` ,x i q t p G p x i q ´ y q 7: x i ` 1 “ xi ´ α ˜ v x i 8: end for
Revised: [Equation] ´ y q 7: x i [Equation] 1 “ x

Revised: [Equation]

Revised: ´

Revised: [Table]

Revised: end for

Revised: Other variations of the tweak are also possible.

Revised: For example, the negation check in Step 2 could be performed after a convergence criterion is satisﬁed, and if a lower objective is achieved by negating the latent code, then the gradient descent can be continued again until a convergence criterion is again satisﬁed.

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: For our analysis, we consider a fully-connected generative network G : R k Ñ R n with Gaussian weights and no bias terms.
Revised: For our analysis, we consider a fully-connected generative network G : R k Ñ R n with Gaussian weights and no bias terms.

Original: Speciﬁcally, we assume that the weights W i are independently and identically distributed as N p 0 , 2 { n i q , but do not require them to be independent across layers.
Revised: Speciﬁcally, we assume that the weights W i are independently and identically distributed as N p 0 , 2 { n i q , but do not require them to be independent across layers.

Original: Moreover, we assume that the network is sufﬁciently expansive :
Revised: Moreover, we assume that the network is sufﬁciently expansive :

Original: Expansivity condition.
Revised: Expansivity condition.

Original: We say that the expansivity condition with constant (cid:15) ą 0 holds if
Revised: We say that the expansivity condition with constant (cid:15) ą 0 holds if

Original: [Equation]
Revised: [Equation]

Original: for all i, where c is a particular numerical constant.
Revised: where c is a particular numerical constant.

Original: In a real-world generative network the weights are learned from training data, and are not drawn from a Gaussian distribution.
Revised: In a real-world generative network the weights are learned from training data, and are not drawn from a Gaussian distribution.

Original: Nonetheless, the motivation for selecting Gaussian weights for our analysis is as follows:
Revised: Nonetheless, the motivation for selecting Gaussian weights for our analysis is as follows:

Original: The empirical distribution of weights from deep neural networks often have statistics consistent with Gaussians.
Revised: The empirical distribution of weights from deep neural networks often have statistics consistent with Gaussians.

Original: AlexNet is a concrete example (Arora et al., 2015).
Revised: AlexNet is a concrete example (Arora et al., 2015).

Original: The ﬁeld of theoretical analysis of recovery guarantees for deep learning is nascent, and Gaussian networks can permit theoretical results because of well developed theories for random matrices.
Revised: The ﬁeld of theoretical analysis of recovery guarantees for deep learning is nascent, and Gaussian networks can permit theoretical results because of well developed theories for random matrices.

Revised: It is not clear which non-Gaussian distribution for weights is superior from the joint perspective of realism and analytical tractability.

Revised: Truly random nets, such as in the Deep Image Prior (Ulyanov et al., 2017), are increasingly becoming of practical relevance.

Revised: Thus, theoretical advances on random nets is of independent interest.

Original: We are now ready to state our main result.
Revised: We are now ready to state our main result.

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider a network with the weights in the i -th layer, W i P R n i ˆ n i ´ 1 , i.i.d.
Revised: Consider a network with the weights in the i -th layer, W i P R n i ˆ n i ´ 1 , i.i.d.

Original: N p 0 , 2 { n i q distributed, and suppose that the network satisﬁes the expansivity condition for some (cid:15) ď K { d 90 .
Revised: N p 0 , 2 { n i q distributed, and suppose that the network satisﬁes the expansivity condition for some (cid:15) ď K { d 90 .

Original: Also, suppose that the noise variance obeys
Revised: Also, suppose that the noise variance obeys

Original: [Equation]
Revised: [Equation]

Original: Consider the iterates of Algorithm 1 with stepsize α “ K 4 1 d 2 .
Revised: Consider the iterates of Algorithm 1 with stepsize α “ K 4 1 d 2 .

Original: Then, there exists a number of steps
Revised: Then, there exists a number of steps

Original: N upper bounded by
Revised: N upper bounded by

Original: [Equation]
Revised: [Equation]

Original: such that after N steps, the iterates of Algorithm 1 obey
Revised: such that after N steps, the iterates of Algorithm 1 obey

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 ´ 2 e ´ 2 k log n
Revised: for all i ě N

Original: [Equation]
Revised: [Equation]

Original:  ř di “
Revised: 2) with probability at least 1 ´ 2 e ´ 2 k log n ´ ř di “

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: e ´ K 
Revised: e ´ K 7 n

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: q k .
Revised: q k .

Original: Here, K 1 , K 2 , .. are numerical constants, and x 0 is the initial point in the optimization.
Revised: Here, K 1 , K 2 , .. are numerical constants, and x 0 is the initial point in the optimization.

Original: The error term in the bound (2) consists of two terms—the ﬁrst is controlled by (cid:15) , and the second depends on the noise.
Revised: The error term in the bound (2) consists of two terms—the ﬁrst is controlled by (cid:15) , and the second depends on the noise.

Original: The ﬁrst term is negligible if (cid:15) is chosen sufﬁciently small, but that comes at the expense of the expansivity condition being more stringent.
Revised: The ﬁrst term is negligible if (cid:15) is chosen sufﬁciently small, but that comes at the expense of the expansivity condition being more stringent.

Original: The second term in the bound (2) is more interesting and controls the effect of noise.
Revised: The second term in the bound (2) is more interesting and controls the effect of noise.

Original: Speciﬁcally, for (cid:15) sufﬁciently small, our result guarantees that after sufﬁciently many iterations,
Revised: Speciﬁcally, for (cid:15) sufﬁciently small, our result guarantees that after sufﬁciently many iterations,

Original: [Equation]
Revised: [Equation]

Original: where the notation À absorbs a factor logarithmic in n and polynomial in d .
Revised: where the notation À absorbs a factor logarithmic in n and polynomial in d .

Original: One can show that G is
Revised: One can show that G is

Original: Lipschitz in a region around x ˚ 1 ,
Revised: Lipschitz in a region around x ˚ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Thus, the theorem guarantees that our algorithm yields the denoising rate of σ 2 k { n , and, as a consequence, denoising based on a generative deep prior provably reduces the energy of the noise in the original image by a factor of k { n .
Revised: Thus, the theorem guarantees that our algorithm yields the denoising rate of σ 2 k { n , and, as a consequence, denoising based on a generative deep prior provably reduces the energy of the noise in the original image by a factor of k { n .

Original: We note that the intention of this paper is to show rate-optimality of recovery with respect to the noise power, the latent code dimensionality, and the signal dimensionality.
Revised: We note that the intention of this paper is to show rate-optimality of recovery with respect to the noise power, the latent code dimensionality, and the signal dimensionality.

Original: As a result, no attempt was made to establish optimal bounds with respect to the scaling of constants or to powers of d .
Revised: As a result, no attempt was made to establish optimal bounds with respect to the scaling of constants or to powers of d .

Original: The bounds provided in the theorem are highly conservative in the constants and dependency on the number of layers, d , in order to keep the proof as simple as possible.
Revised: The bounds provided in the theorem are highly conservative in the constants and dependency on the number of layers, d , in order to keep the proof as simple as possible.

Original: Numerical experiments shown later reveal that the parameter range for successful denoising are much broader than the constants suggest.
Revised: Numerical experiments shown later reveal that the parameter range for successful denoising are much broader than the constants suggest.

Original: As this result is the ﬁrst of its kind for rigorous analysis of denoising performance by deep generative networks, we anticipate the results can be improved in future research, as has happened for other problems, such as sparsity-based compressed sensing and phase retrieval.
Revised: As this result is the ﬁrst of its kind for rigorous analysis of denoising performance by deep generative networks, we anticipate the results can be improved in future research, as has happened for other problems, such as sparsity-based compressed sensing and phase retrieval.

Original: 4. T HE W EIGHT D ISTRIBUTION C ONDITION (WDC)
Revised: 4. T HE W EIGHT D ISTRIBUTION C ONDITION (WDC)

Original: To prove our main result, we make use of a deterministic condition on G , called the Weight Distribution Condition (WDC), and then show that Gaussian W i , as given by the statement of Theoremare such that W i {? 2 satisﬁes the WDC with the appropriate probability for all i , provided the expansivity condition holds.
Revised: To prove our main result, we make use of a deterministic condition on G , called the Weight Distribution Condition (WDC), and then show that Gaussian W i , as given by the statement of Theoremare such that W i {? 2 satisﬁes the WDC with the appropriate probability for all i , provided the expansivity condition holds.

Original: Our main result, Theorem 1, continues to hold for any weight matrices such that W i {?
Revised: Our main result, Theorem 1, continues to hold for any weight matrices such that W i {?

Original: 2 satisfy the WDC.
Revised: 2 satisfy the WDC.

Original: The condition is on the spatial arrangement of the network weights within each layer.
Revised: The condition is on the spatial arrangement of the network weights within each layer.

Original: We say that the matrix W P R n ˆ k satisﬁes the Weight Distribution Condition with constant (cid:15) if for all nonzero x, y P R k ,››› [Equation] w i
Revised: We say that the matrix W P R n ˆ k satisﬁes the Weight Distribution Condition with constant (cid:15) if for all nonzero x, y P R k ,

Original: [Equation]
Revised: [Equation]

Original: ď
Revised: n ÿ

Original: [Equation]
Revised: [Equation]

Original: [-, with Q x,y “ π ´ θ 0 
-]
Revised: [-, with Q x,y “ π ´ θ 0 
-]

Original: [-π I k ` sin θ 0 
 π M ˆ x Ø ˆ y , (3)-] where w i P R k is the i th row of W ;
Revised: [-π I k ` sin θ 0 
 π M ˆ x Ø ˆ y , (3)-] where w i P R k is the i th row of W ;

Original: M ˆ x Ø ˆ y P R k ˆ k is the matrix 2 such that ˆ x ÞÑ ˆ y , ˆ y ÞÑ ˆ x , and z ÞÑ 0 for all z P span pt x, y uq K ; ˆ x “ x {} x } 

Revised: M ˆ x Ø ˆ y P R k ˆ k is the matrix 2 such that ˆ x ÞÑ ˆ y , ˆ y ÞÑ ˆ x , and z ÞÑ 0 for all z P span pt x, y uq K ; ˆ x “ x {} x }

Original: and ˆ y “ y {} y } 2 ; θ 

Revised: and ˆ y “ y {} y } 2 ; θ

Original: “ = p x, y q ; and 1 S is the indicator function on S .
Revised: “ = p x, y q ; and 1 S is the indicator function on S .

Original: The norm in the left hand side of (3) is the spectral norm.
Revised: The norm in the left hand side of (3) is the spectral norm.

Original: Note that an elementary calculation 3 gives that Q x,y “ E r ř ni “ 1 1 (cid:104) w i ,x (cid:105) ą 0 1 (cid:104)
Revised: Note that an elementary calculation 3 gives that Q x,y “ E [Equation]

Original: [-w i ,y (cid:105) ą 
-]
Revised: [-w i ,y (cid:105) ą 
-][Equation]

Original: ¨ w i w ti s for w i „ 
Revised: for w i „ N p [Equation]

Original: N p 0 , I k { n q .
Revised: [Equation] k { n q .

Original: As the rows w i correspond to the neural network weights of the i th neuron in a layer given by W , the WDC provides a deterministic property under which the set of neuron weights within the layer given by W are distributed approximately like a Gaussian.
Revised: As the rows w i correspond to the neural network weights of the i th neuron in a layer given by W , the WDC provides a deterministic property under which the set of neuron weights within the layer given by W are distributed approximately like a Gaussian.

Original: The WDC could also be interpreted as a deterministic property under which the neuron weights are distributed approximately like a uniform random variable on a sphere of a particular radius.
Revised: The WDC could also be interpreted as a deterministic property under which the neuron weights are distributed approximately like a uniform random variable on a sphere of a particular radius.

Original: Note that if x “ y , Q x,y is an isometry up to a factor of 1 { 2 .
Revised: Note that if x “ y , Q x,y is an isometry up to a factor of 1 { 2 .

Original:  A PPLICATIONS TO C OMPRESSED S ENSING
Revised:  A PPLICATIONS TO C OMPRESSED S ENSING

Original: In this section we brieﬂy discuss another important scenario to which our results apply to, namely regularizing inverse problems using deep generative priors.
Revised: In this section we brieﬂy discuss another important scenario to which our results apply to, namely regularizing inverse problems using deep generative priors.

Original: Approaches that regularize inverse problems using deep generative models (Bora et al., 2017) have empirically been shown to improve over sparsity-based approaches, see (Lucas et al., 2018) for a review for applications in imaging, ¨ ˝ cos θsin θ0 sin θ´ cos θ0 0 k ´˛ ‚ R , where ´ “ ˆ respectively.
Revised: Approaches that regularize inverse problems using deep generative models (Bora et al., 2017) have empirically been shown to improve over sparsity-based approaches, see (Lucas et al., 2018) for a review for applications in imaging, and (Mardani et al., 2017) for an application in Magnetic Resonance Imaging showing a signiﬁcant performance improvement over conventional methods.

Original: To do this calculation, take x “ e 1 and y “ cos θ 0 ¨ e 1 ` sin θ 0 ¨ e 2 without loss of generality.

Original: Then each entry of the matrix can be determined analytically by an integral that factors in polar coordinates.

Original: and (Mardani et al., 2017) for an application in Magnetic Resonance Imaging showing a signiﬁcant performance improvement over conventional methods.

Original: Consider an inverse problem, where the goal is to reconstruct an unknown vector y ˚ P R n from m ă n noisy linear measurements:
Revised: Consider an inverse problem, where the goal is to reconstruct an unknown vector y ˚ P R n from m ă n noisy linear measurements:

Original: [Equation]
Revised: [Equation]

Original: where A P R m ˆ n is called the measurement matrix and η is zero mean Gaussian noise with covariance matrix σ 2 { nI , as before.
Revised: where A P R m ˆ n is called the measurement matrix and η is zero mean Gaussian noise with covariance matrix σ 2 { nI , as before.

Original: As before, assume that y ˚ lies in the range of a generative prior G , i.e., y ˚ “ G p x ˚ q for some x ˚ .
Revised: As before, assume that y ˚ lies in the range of a generative prior G , i.e., y ˚ “ G p x ˚ q for some x ˚ .

Original: As a way to recover x ˚ , consider minimizing the empirical risk objective f p x q “ 12 } AG p x q ´ z } , using Algorithm 1, with Step 6 substituted by ˜ v x
Revised: As a way to recover x ˚ , consider minimizing the empirical risk objective f p x q “ 12 } AG p x q ´ z } , using Algorithm 1, with Step 6 substituted by ˜ [Equation]

Original: i “ p A Π 1 i “ d W i, ` ,x i q t p AG
Revised: Π [Equation] i, ` ,x i q t p AG

Original: p x i q ´ y q , to account for the fact that measurements were taken with the matrix A .
Revised: p x i q ´ y q , to account for the fact that measurements were taken with the matrix A .

Original: Suppose that A is a random projection matrix, for concreteness assume that A has i.i.d.
Revised: Suppose that A is a random projection matrix, for concreteness assume that A has i.i.d.

Original: Gaussian entries with variance 1 { m .
Revised: Gaussian entries with variance 1 { m .

Original: One could prove an analogous result as Theorem 1, but with ω “ b 18 σ 2 km log p n d 1 n d ´ 1. . .
Revised: One could prove an analogous result as Theorem 1, but with ω “ b 18 σ 2 k m log p n d 1 n d ´ 1. . .

Original: n d q , (note that n has been replaced by m ).
Revised: n d q , (note that n has been replaced by m ).

Original: This extension shows that, provided (cid:15) is chosen sufﬁciently small, that our algorithm yields an iterate x i obeying
Revised: This extension shows that, provided (cid:15) is chosen sufﬁciently small, that our algorithm yields an iterate x i obeying

Original: [Equation]
Revised: [Equation]

Original: where again À absorbs factors logarithmic in the n i ’s, and polynomial in d .
Revised: where again À absorbs factors logarithmic in the n i ’s, and polynomial in d .

Original: Proving this result would be analogous to the proof of Theorem 1, but with the additional assumption that the sensing matrix A acts like an isometry on the union of the ranges of Π 1 i “
Revised: Proving this result would be analogous to the proof of Theorem 1, but with the additional assumption that the sensing matrix A acts like an isometry on the union of the ranges of Π 1 i “

Original: d W i, ` ,x i , analogous to the proof in (Hand & Voroninski, 2018).
Revised: d W i, ` ,x i , analogous to the proof in (Hand & Voroninski, 2018).

Original: This extension of our result shows that Algorithm 1 enables solving inverse problems under noise efﬁciently, and quantiﬁes the effect of the noise. 
Revised: This extension of our result shows that Algorithm 1 enables solving inverse problems under noise efﬁciently, and quantiﬁes the effect of the noise. [Equation]

Original:  We hasten to add that the paper (Bora et al., 2017) also derived an error bound for minimizing empirical loss.
Revised: [Equation] We hasten to add that the paper (Bora et al., 2017) also derived an error bound for minimizing empirical loss.

Original: However, the corresponding result (for example Lemma 4.3) differs in two important aspects to our result.
Revised: However, the corresponding result (for example Lemma 4.3) differs in two important aspects to our result.

Original: First, the result in (Bora et al., 2017) only makes a statement about the minimizer of the empirical loss and does not provide justiﬁcation that an algorithm can efﬁciently ﬁnd a point near the global minimizer.
Revised: First, the result in (Bora et al., 2017) only makes a statement about the minimizer of the empirical loss and does not provide justiﬁcation that an algorithm can efﬁciently ﬁnd a point near the global minimizer.

Original: As the program is non-convex, and as non-convex optimization is NP-hard in general, the empirical loss could have local minima at which algorithms get stuck.
Revised: As the program is non-convex, and as non-convex optimization is NP-hard in general, the empirical loss could have local minima at which algorithms get stuck.

Original: In contrast, the present paper presents a speciﬁc practical algorithm and proves that it ﬁnds a solution near the global optimizer regardless of initialization.
Revised: In contrast, the present paper presents a speciﬁc practical algorithm and proves that it ﬁnds a solution near the global optimizer regardless of initialization.

Original: Second, the result in (Bora et al., 2017) considers arbitrary noise η and thus can not assert denoising performance.
Revised: Second, the result in (Bora et al., 2017) considers arbitrary noise η and thus can not assert denoising performance.

Original: In contrast, we consider a random model for the noise, and show the denoising behavior that the resulting error is no more than O p k { n q , as opposed to } η } 2 « O p 1 q , which is what we would get from direct application of the result in (Bora et al., 2017).
Revised: In contrast, we consider a random model for the noise, and show the denoising behavior that the resulting error is no more than O p k { n q , as opposed to } η } 2 « O p 1 q , which is what we would get from direct application of the result in (Bora et al., 2017).

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: In this section we provide experimental evidence that corroborates our theoretical claims that denoising with deep priors achieves a denoising rate proportional to σ 2 k { n .
Revised: In this section we provide experimental evidence that corroborates our theoretical claims that denoising with deep priors achieves a denoising rate proportional to σ 2 k { n .

Original: We consider both a synthetic, random prior, as studied theoretically in the paper, as well as a prior learned from data.
Revised: We consider both a synthetic, random prior, as studied theoretically in the paper, as well as a prior learned from data.

Original: All our results are reproducible with the code provided in the supplement.
Revised: All our results are reproducible with the code provided in the supplement.

Original: 6. D ENOISING WITH A SYNTHETIC PRIOR
Revised: 6. D ENOISING WITH A SYNTHETIC PRIOR

Original: We start with a synthetic generative network prior with ReLu-activation functions, and draw its weights independently from a Gaussian distribution.
Revised: We start with a synthetic generative network prior with ReLu-activation functions, and draw its weights independently from a Gaussian distribution.

Original: We consider a two-layer network with n “ 1500 neurons in the output layer, 500 in the middle layer, and vary the number of input neurons, k , and the noise level, σ .
Revised: We consider a two-layer network with n “ 1500 neurons in the output layer, 500 in the middle layer, and vary the number of input neurons, k , and the noise level, σ .

Original: We next present simulations showing that if k is sufﬁciently small, our algorithm achieves a denoising rate proportional to σk { n as guaranteed by our theory.
Revised: We next present simulations showing that if k is sufﬁciently small, our algorithm achieves a denoising rate proportional to σk { n as guaranteed by our theory.

Original: Towards this goal, we generate Gaussian inputs x ˚ to the network and observe the noisy image y “ G p x ˚ q ` η , η „ N p 0 , σ 2 { nI q .
Revised: Towards this goal, we generate Gaussian inputs x ˚ to the network and observe the noisy image y “ G p x ˚ q ` η , η „ N p 0 , σ 2 { nI q .

Original: From the noisy image, we ﬁrst obtain an estimate ˆ x of the latent representation by running Algorithm 1 until convergence, and second we obtain an estimate of the image as ˆ y “ G p ˆ x q .
Revised: From the noisy image, we ﬁrst obtain an estimate ˆ x of the latent representation by running Algorithm 1 until convergence, and second we obtain an estimate of the image as ˆ y “ G p ˆ x q .

Original: In the left and middle panel of Figure 3, we depict the normalized meannoisy denoised noise variance 0. squared error of the latent representation, MSE p ˆ x, x ˚ q , and the mean squared error in the image domain, MSE p G p ˆ x q , G p x ˚ qq , where we deﬁned MSE p z, z 1 q “ } z ´ z 1 } 2 .
Revised: In the left and middle panel of Figure 3, we depict the normalized mean squared error of the latent representation, MSE p ˆ x, x ˚ q , and the mean squared error in the image domain, MSE p G p ˆ x q , G p x ˚ qq , where we deﬁned MSE p z, z 1 q “ } z ´ z 1 } 2 .

Original: For the left panel, we ﬁx the noise variance to σ 2 “ 0 . 25 , and vary k , and for the middle panel we ﬁx k “ 50 and vary the noise variance.
Revised: For the left panel, we ﬁx the noise variance to σ 2 “ 0 . 25 , and vary k , and for the middle panel we ﬁx k “ 50 and vary the noise variance.

Original: The results show that, if the network is sufﬁciently expansive, guaranteed by k being sufﬁciently small, then in the noiseless case ( σ 2 “ 0 ), the latent representation and image are perfectly recovered.
Revised: The results show that, if the network is sufﬁciently expansive, guaranteed by k being sufﬁciently small, then in the noiseless case ( σ 2 “ 0 ), the latent representation and image are perfectly recovered.

Original: In the noisy case, we achieve a MSE proportional to σ 2 k { n , both in the representation and image domains.
Revised: In the noisy case, we achieve a MSE proportional to σ 2 k { n , both in the representation and image domains.

Original: We also observed that for the problem instances considered here, the negation trick in step 3-4 of
Revised: We also observed that for the problem instances considered here, the negation trick in step 3-4 of

Original: Algorithm 1 is often not necessary, in that even without that step the algorithm typically converges to the global minimum.
Revised: Algorithm 1 is often not necessary, in that even without that step the algorithm typically converges to the global minimum.

Original: Having said this, in general the negation step is necessary, since there exist problem instances that have a local minimum opposite of x ˚ .
Revised: Having said this, in general the negation step is necessary, since there exist problem instances that have a local minimum opposite of x ˚ .

Original: 6. D ENOISING WITH A LEARNED PRIOR
Revised: 6. D ENOISING WITH A LEARNED PRIOR

Original: We next consider a prior learned from data.
Revised: We next consider a prior learned from data.

Original: Technically, for such a prior our theory does not apply since we assume the weights to be chosen at random.
Revised: Technically, for such a prior our theory does not apply since we assume the weights to be chosen at random.

Original: However, the numerical results presented in this section show that even for the learned prior we achieve the rate predicted by our theory pertaining to a random prior.
Revised: However, the numerical results presented in this section show that even for the learned prior we achieve the rate predicted by our theory pertaining to a random prior.

Original: Towards this goal, we consider a fully-connected autoencoder parameterized by k , consisting of an decoder and encoder with ReLu activation functions and fully connected layers.
Revised: Towards this goal, we consider a fully-connected autoencoder parameterized by k , consisting of an decoder and encoder with ReLu activation functions and fully connected layers.

Original: We choose the number of neurons in the three layers of the encoder as 784 , 400 , k , and those of the decoder as k, 400 , 784 .
Revised: We choose the number of neurons in the three layers of the encoder as 784 , 400 , k , and those of the decoder as k, 400 , 784 .

Original: We set k “ 10 and k “ 20 to obtain two different autoencoders.
Revised: We set k “ 10 and k “ 20 to obtain two different autoencoders.

Original: We train both autoencoders on the MNIST (Lecun et al., 1998) training set .
Revised: We train both autoencoders on the MNIST (Lecun et al., 1998) training set .

Original: We then take an image y ˚ from the MNIST test set , add Gaussian noise to it, and denoise it using our method based on the learned decoder-network G for k “ 10 and k “ 20 . 
Revised: We then take an image y ˚ from the MNIST test set , add Gaussian noise to it, and denoise it using our method based on the learned decoder-network G for k “ 10 and k “ 20 . Speciﬁcally, we estimate [Figure]

Original: Speciﬁcally, we estimate the latent representation ˆ x by running Algorithm 1, and then set ˆ y “ G p ˆ x q .
Revised: [Figure] [Figure] [Figure] the latent representation ˆ x by running Algorithm 1, and then set ˆ y “ G p ˆ x q .

Original: See Figure 2 for a few examples demonstrating the performance of our approach for different noise levels.
Revised: See Figure 2 for a few examples demonstrating the performance of our approach for different noise levels.

Original: We next show that this achieves a mean squared error (MSE) proportional to σ 2 k { n , as suggested by our theory which applies for decoders with random weights.
Revised: We next show that this achieves a mean squared error (MSE) proportional to σ 2 k { n , as suggested by our theory which applies for decoders with random weights.

Original: We add noise to the images with noise variance ranging from σ 2 “ 0 to σ 2 “ 6 .
Revised: We add noise to the images with noise variance ranging from σ 2 “ 0 to σ 2 “ 6 .

Original: In the right panel of Figure 3 we show the MSE in the image domain, MSE p G p ˆ x q , G p x ˚ qq , averaged over a number of images for the learned decoders with k “ 10 and k “ 20 .
Revised: In the right panel of Figure 3 we show the MSE in the image domain, MSE p G p ˆ x q , G p x ˚ qq , averaged over a number of images for the learned decoders with k “ 10 and k “ 20 .

Original: We observe an interesting tradeoff: The decoder with k “ 10 has fewer parameters, and thus does not represent the digits as well, therefore the MSE is larger than that for k “ 20 for the noiseless case (i.e., for σ “ 0 ).
Revised: We observe an interesting tradeoff: The decoder with k “ 10 has fewer parameters, and thus does not represent the digits as well, therefore the MSE is larger than that for k “ 20 for the noiseless case (i e., for σ “ 0 ).

Original: On the other hand, the smaller number of parameters results in a better denoising rate (by about a factor of two), corresponding to the steeper slope of the
Revised: On the other hand, the smaller number of parameters results in a better denoising rate (by about a factor of two), corresponding to the steeper slope of the

Original: MSE as a function of the noise variance, σ 2 .
Revised: MSE as a function of the noise variance, σ 2 .

Original: A P ROOFS
Revised: A P ROOFS

Original: In this section we prove our main result, Theorem 1.
Revised: In this section we prove our main result, Theorem 1.

Original: Instead of proving Theorem 1 as stated, we will prove the following equivalent rescaled statement for when W i have i.i.d.
Revised: Instead of proving Theorem 1 as stated, we will prove the following equivalent rescaled statement for when W i have i.i.d.

Original: N p 0 , 1 { n i q entries.
Revised: N p 0 , 1 { n i q entries.

Original: Because of this rescaling, G p x q scales like 2 ´ d { 2 } x } , the noise ω is assumed to scale like 2 ´ d { 2 , ∇ f scales like 2 d , and α scales like 2 d .
Revised: Because of this rescaling, G p x q scales like 2 ´ d { 2 } x } , the noise ω is assumed to scale like 2 ´ d { 2 , ∇ f scales like 2 d , and α scales like 2 d .

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider a network with the weights in the i -th layer, W i P R n i ˆ n i ´ 1 , i.i.d.
Revised: Consider a network with the weights in the i -th layer, W i P R n i ˆ n i ´ 1 , i.i.d.

Original: N p 0 , 1 { n i q distributed, and suppose that the network satisﬁes the expansivity condition for some (cid:15) ď K { d 90 .
Revised: N p 0 , 1 { n i q distributed, and suppose that the network satisﬁes the expansivity condition for some (cid:15) ď K { d 90 .

Original: Also, suppose that the noise variance obeys
Revised: Also, suppose that the noise variance obeys

Original: [Equation]
Revised: [Equation]

Original: Consider the iterates of Algorithm 1 with stepsize α “ K 4 2 d d 2 .
Revised: Consider the iterates of Algorithm 1 with stepsize α “ K 4 2 d d 2 .

Original: Then, there exists a number of steps
Revised: Then, there exists a number of steps

Original: N upper bounded by
Revised: N upper bounded by

Original: [Equation]
Revised: [Equation]

Original: such that after N steps, the iterates of Algorithm 1 obey
Revised: such that after N steps, the iterates of Algorithm 1 obey

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 ´ 2 e
Revised: with probability at least 1 ´ 2 e

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q k .
Revised: q k .

Original: Here, K 1 , K 2 , .. are numerical constants, and x 0 is the initial point in the optimization.
Revised: Here, K 1 , K 2 , .. are numerical constants, and x 0 is the initial point in the optimization.

Original: As mentioned in Section 4.1, our proof makes use of a deterministic condition, called the Weight
Revised: As mentioned in Section 4.1, our proof makes use of a deterministic condition, called the Weight

Original: Distribution Condition (WDC), formally deﬁned in Section 4.1.
Revised: Distribution Condition (WDC), formally deﬁned in Section 4.1.

Original: The following proposition establishes that the expansivity condition ensures that the WDC holds:
Revised: The following proposition establishes that the expansivity condition ensures that the WDC holds:

Original: Lemma 3 (Lemma 9 in (Hand & Voroninski, 2018)) .
Revised: Lemma 3 (Lemma 9 in (Hand & Voroninski, 2018)) .

Original: Fix (cid:15) P p 0 , 1 q .
Revised: Fix (cid:15) P p 0 , 1 q .

Original: If the entires of W i P R n i ˆ n i ´are i.i.d.
Revised: If the entires of W i P R n i ˆ n i ´are i.i.d.

Original: N p 0 , 1 { n i q and the expansivity condition n i ą c(cid:15) ´
Revised: N p 0 , 1 { n i q and the expansivity condition n i ą c(cid:15) ´

Original: 2 log p 1 { (cid:15) q n i ´ 1 log n i ´ 1 holds, then W i satisﬁes the WDC with constant (cid:15) with probability at least 1 ´ 8 n i e ´ K(cid:15) 

Revised: 2 log p 1 { (cid:15) q n i ´ 1 log n i ´ 1 holds, then W i satisﬁes the WDC with constant (cid:15) with probability at least 1 ´ 8 n i e ´ K(cid:15)

Original: n i ´ 1 .
Revised: n i ´ 1 .

Original: Here, c and K are numerical constants.
Revised: Here, c and K are numerical constants.

Original: We note that the form of dependence of n i on (cid:15) can be read off the proofs of Lemma 10 in (Hand & Voroninski, 2018).
Revised: We note that the form of dependence of n i on (cid:15) can be read off the proofs of Lemma 10 in (Hand & Voroninski, 2018).

Original: It follows from Lemma 3, that the WDC holds for all W i with probability at least
Revised: It follows from Lemma 3, that the WDC holds for all W i with probability at least

Original: [Equation]
Revised: [Equation]

Original: In the remainder of the proof we work on the event that the WDC holds for all W i .
Revised: In the remainder of the proof we work on the event that the WDC holds for all W i .

Original: A. P RELIMINARIES
Revised: A. P RELIMINARIES

Original: Recall that the goal of our algorithm is to minimize the empirical risk objective
Revised: Recall that the goal of our algorithm is to minimize the empirical risk objective

Original: [Equation]
Revised: [Equation]

Original: where y : “ G p x ˚ q ` η , with η „ N p 0 , σ 2 { nI q .
Revised: where y : “ G p x ˚ q ` η , with η „ N p 0 , σ 2 { nI q .

Original: Our results rely on the fact that outside of two balls around x “ x ˚ and x “ ´ ρ d x ˚ , with ρ d a constant deﬁned below, the direction chosen by the algorithm is a descent direction, with high probability.
Revised: Our results rely on the fact that outside of two balls around x “ x ˚ and x “ ´ ρ d x ˚ , with ρ d a constant deﬁned below, the direction chosen by the algorithm is a descent direction, with high probability.

Original: Towards this goal, we use a concentration argument, similar to the arguments used in (Hand & Voroninski, 2018).
Revised: Towards this goal, we use a concentration argument, similar to the arguments used in (Hand & Voroninski, 2018).

Original: First, deﬁne Λ x : “ Π 1 i “ d W i, ` ,x (with W i, ` ,x deﬁned in Section 3) for notational convenience, and note that the step direction of our algorithm can be written as
Revised: First, deﬁne Λ x : “ Π 1 i “ d W i, ` ,x (with W i, ` ,x deﬁned in Section 3) for notational convenience, and note that the step direction of our algorithm can be written as

Original: [Equation]
Revised: [Equation]

Original: Note that at points x where G (and hence f ) is differentiable, we have that ˜ v x “ ∇ f p x q .
Revised: Note that at points x where G (and hence f ) is differentiable, we have that ˜ v x “ ∇ f p x q .

Original: The proof is based on showing that ˜ v x concentrates around a particular h x P R k , deﬁned below, that is a continuous function of nonzero x, x ˚ and is zero only at x “ x ˚ and x “ ´ ρ d x ˚ .
Revised: The proof is based on showing that ˜ v x concentrates around a particular h x P R k , deﬁned below, that is a continuous function of nonzero x, x ˚ and is zero only at x “ x ˚ and x “ ´ ρ d x ˚ .

Original: The deﬁnition of h x depends on a function that is helpful for controlling how the operator x ÞÑ W ` ,x x distorts angles, deﬁned as:
Revised: The deﬁnition of h x depends on a function that is helpful for controlling how the operator x ÞÑ W ` ,x x distorts angles, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: With this notation, we deﬁne
Revised: With this notation, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where θ 0 “ = p x, x ˚ q and θ i “ g p θ
Revised: where θ 0 “ = p x, x ˚ q and θ i “ g p θ

Original: i ´ 1 q .
Revised: i ´ 1 q .

Original: Note that h x is deterministic and only depends on x , x ˚ , and the number of layers, d .
Revised: Note that h x is deterministic and only depends on x , x ˚ , and the number of layers, d .

Original: In order to bound the deviation of ˜ v x from h x we use the following two lemmas, bounding the deviation controlled by the WDC and the deviation from the noise:
Revised: In order to bound the deviation of ˜ v x from h x we use the following two lemmas, bounding the deviation controlled by the WDC and the deviation from the noise:

Original: Lemma 4 (Lemma 6 in (Hand & Voroninski, 2018)) .
Revised: Lemma 4 (Lemma 6 in (Hand & Voroninski, 2018)) .

Original: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 .
Revised: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 .

Original: Then, for all nonzero x, x ˚ P R k , } v x ´ h x } 2 ď K d
Revised: Then, for all nonzero x, x ˚ P R k , } v x ´ h x } 2 ď K d

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Equation (7) and (8) are Lemma 6 in (Hand & Voroninski, 2018).
Revised: Equation (7) and (8) are Lemma 6 in (Hand & Voroninski, 2018).

Original: Regarding (9), note that the WDC implies that } W i, ` ,x } 2 ď 1 { 2 ` (cid:15) .
Revised: Regarding (9), note that the WDC implies that } W i, ` ,x } 2 ď 1 { 2 ` (cid:15) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: where the last inequalities follow by our assumption on (cid:15) .
Revised: where the last inequalities follow by our assumption on (cid:15) .

Original: Lemma 5.
Revised: Lemma 5.

Original: Suppose the WDC holds with (cid:15)
Revised: Suppose the WDC holds with (cid:15)

Original: ă 1 {p 16 πd 2 q 2 , that any subset of n i ´ 1 rows of W i are linearly independent for each i , and that η „ N p 0 , σ 2 { nI q .
Revised: ă 1 {p 16 πd 2 q 2 , that any subset of n i ´ 1 rows of W i are linearly independent for each i , and that η „ N p 0 , σ 2 { nI q .

Original: Then the event
Revised: Then the event

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: holds with probability at least 1 ´ 2 e ´ 2 k log n .
Revised: holds with probability at least 1 ´ 2 e ´ 2 k log n .

Original: As the cost function f is not differentiable everywhere, we will make use of the generalized subdifferential in order to reference the subgradients at nondifferentiable points.
Revised: As the cost function f is not differentiable everywhere, we will make use of the generalized subdifferential in order to reference the subgradients at nondifferentiable points.

Original: For a Lipschitz function ˜ f deﬁned from a Hilbert space X to R , the Clarke generalized directional derivative of ˜ f at the point x P X in the direction u , denoted by ˜ f
Revised: For a Lipschitz function ˜ f deﬁned from a Hilbert space X to R , the Clarke generalized directional derivative of ˜ f at the point x P X in the direction u , denoted by ˜ f

Original: o p x ; u q , is deﬁned by ˜ f
Revised: o p x ; u q , is deﬁned by ˜ f

Original: o p x ; u q “ lim sup y Ñ x,t Ó 0 ˜ f p y ` tu q´ ˜ f p y q t , and the generalized subdifferential of ˜ f at x , denoted by B ˜ f p x q , is deﬁned by
Revised: o p x ; u q “ lim sup y Ñ x,t Ó 0 ˜ f p y ` tu q´ ˜ f p y q t , and the generalized subdifferential of ˜ f at x , denoted by B ˜ f p x q , is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for all
Revised: for all

Original: [Equation]
Revised: [Equation]

Original: Since f p x q is a piecewise quadratic function, we have
Revised: Since f p x q is a piecewise quadratic function, we have

Original: [Equation]
Revised: [Equation]

Original: (11) where conv denotes the convex hull of the vectors v 1 , . . .
Revised: (11) where conv denotes the convex hull of the vectors v 1 , . . .

Original: , v t , t is the number of quadratic functions adjoint to x , and v i is the gradient of the i -th quadratic function at x .
Revised: , v t , t is the number of quadratic functions adjoint to x , and v i is the gradient of the i -th quadratic function at x .

Original: Lemma 6.
Revised: Lemma 6.

Original: Under the assumption of Lemma 5, and assuming that E noise holds, we have that, for any x ‰ 0 and any v x P B f p x q ,
Revised: Under the assumption of Lemma 5, and assuming that E noise holds, we have that, for any x ‰ 0 and any v x P B f p x q ,

Original: [Equation]
Revised: [Equation]

Original: In particular, this holds for the subgradient v x “ ˜ v x .
Revised: In particular, this holds for the subgradient v x “ ˜ v x .

Original: Proof.
Revised: Proof.

Original: By (11), B f p x q “ conv p v 1 , . . .
Revised: By (11), B f p x q “ conv p v 1 , . . .

Original: v t q for some ﬁnite t , and thus v x “ a 1 v 1 ` . . .
Revised: v t q for some ﬁnite t , and thus v x “ a 1 v 1 ` . . .

Original: a t v t for some a 1 , . . .
Revised: a t v t for some a 1 , . . .

Original: , a t ě 0 , ř i a i “ 1 .
Revised: , a t ě 0 , ř i a i “ 1 .

Original: For each v i , there exists a w such that v i “ lim t Ó 0 ˜ v x ` tw .
Revised: For each v i , there exists a w such that v i “ lim t Ó 0 ˜ v x ` tw .

Original: On the event
Revised: On the event

Original: E noise , we have that for any x ‰ 0 , for any ˜ v x P B f p x q
Revised: E noise , we have that for any x ‰ 0 , for any ˜ v x P B f p x q

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Lemmas 4 and 5 above.
Revised: where the last inequality follows from Lemmas 4 and 5 above.

Original: The proof is concluded by appealing to the continuity of h x with respect to nonzero x , and by noting that
Revised: The proof is concluded by appealing to the continuity of h x with respect to nonzero x , and by noting that

Original: [Equation]
Revised: [Equation]

Original: where we used the inequality above and that ř i a i “ 1 .
Revised: where we used the inequality above and that ř i a i “ 1 .

Original: We will also need an upper bound on the norm of the step direction of our algorithm:
Revised: We will also need an upper bound on the norm of the step direction of our algorithm:

Original: Lemma 7.
Revised: Lemma 7.

Original: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 and that the event E noise holds with ω ď 2 ´ d { 2 } x ˚ } 8 π .
Revised: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 and that the event E noise holds with ω ď 2 ´ d { 2 } x ˚ } 8 π .

Original: Then, for all x ,
Revised: Then, for all x ,

Original: [Equation]
Revised: [Equation]

Original: where K is a numerical constant.
Revised: where K is a numerical constant.

Original: Proof.
Revised: Proof.

Original: Deﬁne for convenience ζ j “ ś d ´ 

Revised: Deﬁne for convenience ζ j “ ś d ´

Original: i
Revised: i

Original: “ j
Revised: “ j

Original: [Equation]
Revised: [Equation]

Original: ´ ¯ θ j,x,x ˚ π .
Revised: ´ ¯ θ j,x,x ˚ π .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from the deﬁnition of h x and Lemma 6, the third inequality uses | ζ j | ď 1 , and the last inequality uses the assumption ω ď 2 ´ d { 2 } x ˚ } 8 π .
Revised: where the second inequality follows from the deﬁnition of h x and Lemma 6, the third inequality uses | ζ j | ď 1 , and the last inequality uses the assumption ω ď 2 ´ d { 2 } x ˚ } 8 π .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: We are now ready to prove Theorem 2.
Revised: We are now ready to prove Theorem 2.

Original: The logic of the proof is illustrated in Figure 4.
Revised: The logic of the proof is illustrated in Figure 4.

Original: Recall that x
Revised: Recall that x

Original: i is the i th iterate of x as per Algorithm 1.
Revised: i is the i th iterate of x as per Algorithm 1.

Original: We ﬁrst ensure that we can assume throughout that x i is bounded away from zero:
Revised: We ﬁrst ensure that we can assume throughout that x i is bounded away from zero:

Original: Lemma 8.
Revised: Lemma 8.

Original: Suppose that WDC holds with (cid:15) ă 1 {p 16 πd 2 q 

Revised: Suppose that WDC holds with (cid:15) ă 1 {p 16 πd 2 q

Original: and that E noise holds with ω in (10) obeying ω ď 2 ´ d { 2 } x ˚ } 8 π .
Revised: and that E noise holds with ω in (10) obeying ω ď 2 ´ d { 2 } x ˚ } 8 π .

Original: Moreover, suppose that the step size in Algorithm 1 satisﬁes 

Revised: Moreover, suppose that the step size in Algorithm 1 satisﬁes

Original: ă
Revised: ă

Original: α ă K 2 d d 2 , where K is a numerical constant.
Revised: α ă K 2 d d 2 , where K is a numerical constant.

Original: Then, after at most N “ p 38 πK 0 2 d α q 2 steps, we have that for all i ą N that x i R B p 0 , K 0 } x ˚ }q , K 0 “ 132 π .
Revised: Then, after at most N “ p 38 πK 0 2 d α q 2 steps, we have that for all i ą N that x i R B p 0 , K 0 } x ˚ }q , K 0 “ 132 π .

Original: In particular, if α “ K 2 d { d 2 , then N is bounded by a constant times d 4 .
Revised: In particular, if α “ K 2 d { d 2 , then N is bounded by a constant times d 4 .

Original: We can therefore assume throughout this proof that x i R B p 0 , K 0 } x ˚ }q , K 0 “ 132 π .
Revised: We can therefore assume throughout this proof that x i R B p 0 , K 0 } x ˚ }q , K 0 “ 132 π .

Original: We prove
Revised: We prove

Original: Theorem 2 by showing that if } h x } is sufﬁciently large, i.e., if the iterate x i is outside of set
Revised: Theorem 2 by showing that if } h x } is sufﬁciently large, i.e., if the iterate x i is outside of set

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: (13) then the algorithm makes progress in the sense that f p x i ` 1 q´ f p x i q is smaller than a certain negative value.
Revised: (13) then the algorithm makes progress in the sense that f p x i ` 1 q´ f p x i q is smaller than a certain negative value.

Original: The set S β is contained in two balls around x ˚ and ´ ρx ˚ , whose radius is controlled by β :
Revised: The set S β is contained in two balls around x ˚ and ´ ρx ˚ , whose radius is controlled by β :

Original: Lemma 9.
Revised: Lemma 9.

Original: For any β ď
Revised: For any β ď

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, ρ d ą 0 is deﬁned in the proof and obeys ρ d Ñ 1 as d Ñ 8 .
Revised: Here, ρ d ą 0 is deﬁned in the proof and obeys ρ d Ñ 1 as d Ñ 8 .

Original: Note that by the assumption ω
Revised: Note that by the assumption ω

Original: ď
Revised: ď

Original: [Equation]
Revised: [Equation]

Original: and Kd 45 ?
Revised: and Kd 45 ?

Original: (cid:15) ď 1 , our choice of β in (13) obeys β ď
Revised: (cid:15) ď 1 , our choice of β in (13) obeys β ď

Original: [Equation]
Revised: [Equation]

Original: for sufﬁciently small K 1 , K , and thus Lemma 9 yields:
Revised: for sufﬁciently small K 1 , K , and thus Lemma 9 yields:

Original: [Equation]
Revised: [Equation]

Original: were we deﬁne the radius r “ K 2 d 9 ?
Revised: were we deﬁne the radius r “ K 2 d 9 ?

Original: (cid:15) } x ˚ } ` K 3 d 6 ω 2 d { 2 , where K 2 , K 3 are numerical constants.
Revised: (cid:15) } x ˚ } ` K 3 d 6 ω 2 d { 2 , where K 2 , K 3 are numerical constants.

Original: Note that hat the radius r is equal to the right hand side in the error bound (4) in our theorem.
Revised: Note that hat the radius r is equal to the right hand side in the error bound (4) in our theorem.

Original: In order to guarantee that the algorithm converges to a ball around x ˚ , and not to that around ´ ρ d x ˚ , we use the following lemma:
Revised: In order to guarantee that the algorithm converges to a ball around x ˚ , and not to that around ´ ρ d x ˚ , we use the following lemma:

Original: Lemma 10.
Revised: Lemma 10.

Original: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 .
Revised: Suppose that the WDC holds with (cid:15) ă 1 {p 16 πd 2 q 2 .

Original: Moreover suppose that E noise holds, and that ω in the event E noise obeys
Revised: Moreover suppose that E noise holds, and that ω in the event E noise obeys

Original: [Equation]
Revised: [Equation]

Original: , where K 9 ă 1 is a universal constant.
Revised: , where K 9 ă 1 is a universal constant.

Original: Then for any φ d P r ρ d , 1 s , it holds that
Revised: Then for any φ d P r ρ d , 1 s , it holds that

Original: [Equation]
Revised: [Equation]

Original: for all x P B p φ d x ˚ , K 3 d ´ 10 } x ˚ }q and y P B p´ φ d x ˚ , K 3 d ´ 10 } x ˚ }q , where K 3 ă 1 is a universal constant.
Revised: for all x P B p φ d x ˚ , K 3 d ´ 10 } x ˚ }q and y P B p´ φ d x ˚ , K 3 d ´ 10 } x ˚ }q , where K 3 ă 1 is a universal constant.

Original: In order to apply Lemma 10, deﬁne for convenience the two sets:
Revised: In order to apply Lemma 10, deﬁne for convenience the two sets:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the assumption that Kd 45 ?
Revised: By the assumption that Kd 45 ?

Original: (cid:15) ď 1 and ω ď K 1 d ´ 16 2 ´ d { 2 } x ˚ } , we have that for sufﬁciently small K 1 , K ,
Revised: (cid:15) ď 1 and ω ď K 1 d ´ 16 2 ´ d { 2 } x ˚ } , we have that for sufﬁciently small K 1 , K ,

Original: [Equation]
Revised: [Equation]

Original: Thus, the assumptions of Lemma 10 are met, and the lemma implies that for any x P S ´ β and y P S ` β , it holds that f p x q ą f p y q .
Revised: Thus, the assumptions of Lemma 10 are met, and the lemma implies that for any x P S ´ β and y P S ` β , it holds that f p x q ą f p y q .

Original: We now show that the algorithm converges to a point in S ` β .
Revised: We now show that the algorithm converges to a point in S ` β .

Original: This fact and the negation step in our algorithm (line 3-5) establish that the algorithm converges to a point in S ` β if we prove that the objective is nonincreasing with iteration number, which will form the remainder of this proof.
Revised: This fact and the negation step in our algorithm (line 3-5) establish that the algorithm converges to a point in S ` β if we prove that the objective is nonincreasing with iteration number, which will form the remainder of this proof.

Original: Consider i such that x i R S β .
Revised: Consider i such that x i R S β .

Original: By the mean value theorem (Clason, 2017, Theorem 8.13), there is a t P r 0 , 1 s such that for ˆ x i “ x i ´ tα ˜ v x i there is a v ˆ x i P B
Revised: By the mean value theorem (Clason, 2017, Theorem 8.13), there is a t P r 0 , 1 s such that for ˆ x i “ x i ´ tα ˜ v x i there is a v ˆ x i P B

Original: f p ˆ x i q , where B f is the generalized subdifferential of f , obeying
Revised: f p ˆ x i q , where B f is the generalized subdifferential of f , obeying

Original: [Equation]
Revised: [Equation]

Original: In the next subsection, we guarantee that for any t P r 0 , 1 s , v ˆ x i with ˆ x i “ x i ´ tα ˜ v x i is close to ˜ v x i :
Revised: In the next subsection, we guarantee that for any t P r 0 , 1 s , v ˆ x i with ˆ x i “ x i ´ tα ˜ v x i is close to ˜ v x i :

Original: [Equation]
Revised: [Equation]

Original: Applying (17) to (16) yields
Revised: Applying (17) to (16) yields

Original: [Equation]
Revised: [Equation]

Original: where we used that αK 7 d 2 2 d ď 112 , by our assumption on the stepsize α being sufﬁciently small.
Revised: where we used that αK 7 d 2 2 d ď 112 , by our assumption on the stepsize α being sufﬁciently small.

Original: Thus, the maximum number of iterations for which x i R S β is f p x 0 q 12 {p α min i } ˜ v x i } 2 q .
Revised: Thus, the maximum number of iterations for which x i R S β is f p x 0 q 12 {p α min i } ˜ v x i } 2 q .

Original: We next lower-bound } ˜ v x i } .
Revised: We next lower-bound } ˜ v x i } .

Original: We have that on E noise , for all x R S β , with β given by (13).
Revised: We have that on E noise , for all x R S β , with β given by (13).

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows by the deﬁnition of S β and Lemma 6, and the third inequality follows from our deﬁnition of β in (13).
Revised: where the second inequality follows by the deﬁnition of S β and Lemma 6, and the third inequality follows from our deﬁnition of β in (13).

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where we used α “ K 4 2 d d 2 .
Revised: where we used α “ K 4 2 d d 2 .

Original: Hence, there can be at most
Revised: Hence, there can be at most

Original: [Equation]
Revised: [Equation]

Original: which
Revised: which

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: In order to conclude our proof, we remark that once x i is inside a ball of radius r around x ˚ , the iterates do not leave a ball of radius 2 r around x ˚ .
Revised: In order to conclude our proof, we remark that once x i is inside a ball of radius r around x ˚ , the iterates do not leave a ball of radius 2 r around x ˚ .

Original: To see this, note that by (12) and our choice of stepsize,
Revised: To see this, note that by (12) and our choice of stepsize,

Original: [Equation]
Revised: [Equation]

Original: This concludes our proof.
Revised: This concludes our proof.

Original: The remainder of the proof is devoted to prove the lemmas used in this section.
Revised: The remainder of the proof is devoted to prove the lemmas used in this section.

Original: A. P ROOF OF E QUATION (17)
Revised: A. P ROOF OF E QUATION (17)

Original: Our proof relies on h x being Lipschitz, as formalized by the lemma below, which is proven in
Revised: Our proof relies on h x being Lipschitz, as formalized by the lemma below, which is proven in

Original: Section A.9:
Revised: Section A.9:

Original: Lemma 11.
Revised: Lemma 11.

Original: For any x, y R B p 0 , K 0 } x ˚ }q , where K 0 and K 4 are numerical constants,
Revised: For any x, y R B p 0 , K 0 } x ˚ }q , where K 0 and K 4 are numerical constants,

Original: [Equation]
Revised: [Equation]

Original: By Lemma 11, for all t P r 0 , 1 s and i ą N (recall that by Lemma 8, after at most N steps, x i ‰ B p 0 , K 0 } x ˚ }q ):
Revised: By Lemma 11, for all t P r 0 , 1 s and i ą N (recall that by Lemma 8, after at most N steps, x i ‰ B p 0 , K 0 } x ˚ }q ):

Original: [Equation]
Revised: [Equation]

Original: (19) where ˆ x i “ x i ´ tα ˜ v x i .
Revised: (19) where ˆ x i “ x i ´ tα ˜ v x i .

Original: Thus, we have that on E noise , for any v
Revised: Thus, we have that on E noise , for any v

Original: ˆ x i P B
Revised: ˆ x i P B

Original: f p ˆ x i q by Lemma 6, } v ˆ x i ´ ˜ v x i } ď} v ˆ x
Revised: f p ˆ x i q by Lemma 6, } v ˆ x i ´ ˜ v x i } ď} v ˆ x

Original: i ´ h ˆ x i
Revised: i ´ h ˆ x i

Original: } ` } h ˆ x
Revised: } ` } h ˆ x

Original: i ´
Revised: i ´

Original: [Equation]
Revised: [Equation]

Original: x i } `
Revised: x i } `

Original: [Equation]
Revised: [Equation]

Original: h x i ´ ˜
Revised: h x i ´ ˜

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second inequality is from Lemma 6 and Equation 19, and the fourth inequality is from (12) and the assumption
Revised: where the second inequality is from Lemma 6 and Equation 19, and the fourth inequality is from (12) and the assumption

Original: [Equation]
Revised: [Equation]

Original: Combining (20) and (18), we get that
Revised: Combining (20) and (18), we get that

Original: [Equation]
Revised: [Equation]

Original: with the appropriate constants chosen sufﬁciently small.
Revised: with the appropriate constants chosen sufﬁciently small.

Original: This concludes the proof of Equation (17).
Revised: This concludes the proof of Equation (17).

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: First suppose that x i P B p 0 , 2 K 0 } x ˚ }q .
Revised: First suppose that x i P B p 0 , 2 K 0 } x ˚ }q .

Original: We show that after a polynomial number of iterations N , we have that x i ` N R B p 0 , 2 K 0 } x ˚ }q .
Revised: We show that after a polynomial number of iterations N , we have that x i ` N R B p 0 , 2 K 0 } x ˚ }q .

Original: Below, we prove that
Revised: Below, we prove that

Original: [Equation]
Revised: [Equation]

Original: It follows that for any x i P B p 0 , 2 K 0 } x ˚ }q , x i and the next iterate produced by the algorithm, x i ` 1 “ x i ´ α ˜ v x i , form an obtruse triangle.
Revised: It follows that for any x i P B p 0 , 2 K 0 } x ˚ }q , x i and the next iterate produced by the algorithm, x i ` 1 “ x i ´ α ˜ v x i , form an obtruse triangle.

Original: As a consequence,
Revised: As a consequence,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (21).
Revised: where the last inequality follows from (21).

Original: Thus, the norm of the iterates x i will increase until after ` 2 K 0 2 d 16 π α ˘ 2 iterations, we have x i ` N R B p 0 , 2 K 0 } x ˚ }q .
Revised: Thus, the norm of the iterates x i will increase until after ` 2 K 0 2 d 16 π α ˘ 2 iterations, we have x i ` N R B p 0 , 2 K 0 } x ˚ }q .

Original: The proof of the lemma is concluded by showing that
Revised: The proof of the lemma is concluded by showing that

Original: [Equation]
Revised: [Equation]

Original: As a consequence, in a polynomial number N of steps, for each iterate, we have that x i R
Revised: As a consequence, in a polynomial number N of steps, for each iterate, we have that x i R

Original: B p
Revised: B p

Original: We next prove the implication (22).
Revised: We next prove the implication (22).

Original: Consider x i R B p 0 , 2 K 0 } x ˚ }q , and note that
Revised: Consider x i R B p 0 , 2 K 0 } x ˚ }q , and note that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from (12), the third inequality from } x i } ě 2 K 0 } x ˚ } , and ﬁnally the last inequality from our assumption on the stepsize α .
Revised: where the second inequality follows from (12), the third inequality from } x i } ě 2 K 0 } x ˚ } , and ﬁnally the last inequality from our assumption on the stepsize α .

Original: This concludes the proof of (22).
Revised: This concludes the proof of (22).

Original: Proof of (21) : It remains to prove (21).
Revised: Proof of (21) : It remains to prove (21).

Original: We start with proving (cid:104) x, ˜ v x (cid:105) ă 0 .
Revised: We start with proving (cid:104) x, ˜ v x (cid:105) ă 0 .

Original: For brevity of notation, let Λ z “ śi
Revised: For brevity of notation, let Λ z “ śi

Original: [Equation]
Revised: [Equation]

Original: d W i, ` ,z .
Revised: d W i, ` ,z .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality follows from (8) and (9), and the second inequality follows from our assumption on ω .
Revised: The ﬁrst inequality follows from (8) and (9), and the second inequality follows from our assumption on ω .

Original: Therefore, for any x P B p 0 , 116 π } x ˚ }q , (cid:104) x, ˜ v x (cid:105) ă 0 , as desired.
Revised: Therefore, for any x P B p 0 , 116 π } x ˚ }q , (cid:104) x, ˜ v x (cid:105) ă 0 , as desired.

Original: We next show that, for any x P B p 0 , 116 π } x ˚ }q
Revised: We next show that, for any x P B p 0 , 116 π } x ˚ }q

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is from (8) and (9).
Revised: where the second inequality is from (8) and (9).

Original: This concludes the proof of (21).
Revised: This concludes the proof of (21).

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Let Λ x “ Π 1 i “ d W i, ` ,x .
Revised: Let Λ x “ Π 1 i “ d W i, ` ,x .

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: where P Λ x is a projector onto the span of Λ x .
Revised: where P Λ x is a projector onto the span of Λ x .

Original: As a consequence, } P Λ x η } 2 is χ 2 -distributed random variable with k -degrees of freedom scaled by σ { n .
Revised: As a consequence, } P Λ x η } 2 is χ 2 -distributed random variable with k -degrees of freedom scaled by σ { n .

Original: A standard tail bound (see ( ? , p. 43)) yields that, for any β ě k ,
Revised: A standard tail bound (see ( ? , p. 43)) yields that, for any β ě k ,

Original: [Equation]
Revised: [Equation]

Original: Next, we note that by applying Lemmas 13-14 from (Hand & Voroninski, 2018, Proof of Lem. 15)) 4 , with probability one, that the number of different matrices Λ x can be bounded as
Revised: Next, we note that by applying Lemmas 13-14 from (Hand & Voroninski, 2018, Proof of Lem. 15)) 4 , with probability one, that the number of different matrices Λ x can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds for log p 10 q ď k { 4 log p n 1 q .
Revised: where the second inequality holds for log p 10 q ď k { 4 log p n 1 q .

Original: To see this, note that p n d 1 n d ´ 1. . .
Revised: To see this, note that p n d 1 n d ´ 1. . .

Original: n d q k ě 10 d 2 is implied by k p d log p n 1 q ` p d ´ 1 q log p n 2 q ` . . .
Revised: n d q k ě 10 d 2 is implied by k p d log p n 1 q ` p d ´ 1 q log p n 2 q ` . . .

Original: log p n d qq ě kd 2 { 4 log p n 1 q ě d 2 log p 10 q .
Revised: log p n d qq ě kd 2 { 4 log p n 1 q ě d 2 log p 10 q .

Original: Thus, by the union bound,
Revised: Thus, by the union bound,

Original: [Equation]
Revised: [Equation]

Original: where n “ n d .
Revised: where n “ n d .

Original: Recall from (9) that } Λ x } ď 1312 .
Revised: Recall from (9) that } Λ x } ď 1312 .

Original: Combining this inequality with } ¯ q x } 2 ď } Λ x } 2 } P Λ x η } 2 concludes the proof.
Revised: Combining this inequality with } ¯ q x } 2 ď } Λ x } 2 } P Λ x η } 2 concludes the proof.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: We now show that h x is away from zero outside of a neighborhood of x ˚ and ´ ρ d x ˚ .
Revised: We now show that h x is away from zero outside of a neighborhood of x ˚ and ´ ρ d x ˚ .

Original: We prove
Revised: We prove

Original: Lemma 9 by establishing the following:
Revised: Lemma 9 by establishing the following:

Original: Lemma 12.
Revised: Lemma 12.

Original: Suppose 64 d 6 ?
Revised: Suppose 64 d 6 ?

Original: β ď 1 .
Revised: β ď 1 .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where q θ 

Revised: where q θ

Original: “ π and q θ i “ g p q θ i ´ 1 q .
Revised: “ π and q θ i “ g p q θ i ´ 1 q .

Original: If x P S β , then we have that either | θ 0 | ď 32 d 4 β and |} x } 2 ´ } x ˚ } 2 | ď 132 d 6 β } x ˚ }or | θ 

Revised: If x P S β , then we have that either | θ 0 | ď 32 d 4 β and |} x } 2 ´ } x ˚ } 2 | ď 132 d 6 β } x ˚ }or | θ

Original: ´ π | ď 8 πda β and |} x } 2 ´
Revised: ´ π | ď 8 πda β and |} x } 2 ´

Original: [Equation]
Revised: [Equation]

Original: x ˚
Revised: x ˚

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: x ˚ } 2 .
Revised: x ˚ } 2 .

Original: In particular, we have
Revised: In particular, we have

Original: [Equation]
Revised: [Equation]

Original: Additionally, ρ d Ñ 1 as d Ñ 8 .
Revised: Additionally, ρ d Ñ 1 as d Ñ 8 .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let } x ˚ } “ 1 , x ˚ “ e 1 and ˆ x “ r cos θ 0 ¨ e 1 ` r sin θ 0 ¨ e 2 for θ 0 P r 0 , π s .
Revised: Without loss of generality, let } x ˚ } “ 1 , x ˚ “ e 1 and ˆ x “ r cos θ 0 ¨ e 1 ` r sin θ 0 ¨ e 2 for θ 0 P r 0 , π s .

Original: Let x P S β .
Revised: Let x P S β .

Original: First we introduce some notation for convenience.
Revised: First we introduce some notation for convenience.

Original: Let ξ “ d ´ 1 ź i “π ´ θ i
Revised: Let ξ “ d ´ 1 ź i “π ´ θ i

Original: [Equation]
Revised: [Equation]

Original: ÿ i
Revised: ÿ i

Original: [Equation]
Revised: [Equation]

Original: sin
Revised: sin

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ź
Revised: ź

Original: [Equation]
Revised: [Equation]

Original: ´
Revised: ´

Original: [Equation]
Revised: [Equation]

Original: π , r
Revised: π , r

Original: [Equation]
Revised: [Equation]

Original: } x } 2 ,
Revised: } x } 2 ,

Original: M “ max p r, 1 q .
Revised: M “ max p r, 1 q .

Original: Thus, h x “ ´ 12 d ξ ˆ x 0 ` 12 d p r ´ ζ q ˆ x .
Revised: Thus, h x “ ´ 12 d ξ ˆ x 0 ` 12 d p r ´ ζ q ˆ x .

Original: By inspecting the components of h x , we have that x P S β implies
Revised: By inspecting the components of h x , we have that x P S β implies

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we record several properties.
Revised: we record several properties.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: We now establish (28).
Revised: We now establish (28).

Original: Observe 

Revised: Observe

Original: ă g p θ q ď
Revised: ă g p θ q ď

Original: [Equation]
Revised: [Equation]

Original: ˘
Revised: ˘

Original: [Equation]
Revised: [Equation]

Original: “ : ˜ g p θ q for θ P p 0 , π s .
Revised: “ : ˜ g p θ q for θ P p 0 , π s .

Original: As g and ˜ g are monotonic increasing, we have q θ i “ g ˝
Revised: As g and ˜ g are monotonic increasing, we have q θ i “ g ˝

Original: i p q θ 0 q
Revised: i p q θ 0 q

Original: “ g ˝ i p π q ď ˜ g
Revised: “ g ˝ i p π q ď ˜ g

Original: ˝ i p π q “ ` i 3 π ` 1 π ˘ ´ 1 “ 3 π i ` 3 .
Revised: ˝ i p π q “ ` i 3 π ` 1 π ˘ ´ 1 “ 3 π i ` 3 .

Original: Similarly, g p θ q ě p 1 π ` 1 θ q ´ 1 implies that q θ i ě πi ` 1 , establishing (29).
Revised: Similarly, g p θ q ě p 1 π ` 1 θ q ´ 1 implies that q θ i ě πi ` 1 , establishing (29).

Original: We now establish (30).
Revised: We now establish (30).

Original: Using (28) and θ i
Revised: Using (28) and θ i

Original: ď q θ i , we have
Revised: ď q θ i , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality can be established by showing that the ratio of consecutive terms with respect to d is greater for the product in the middle expression than for d ´ 3 .
Revised: where the last inequality can be established by showing that the ratio of consecutive terms with respect to d is greater for the product in the middle expression than for d ´ 3 .

Original: We establish (31) by using the fact that | g 1 p θ q| ď 1 for all θ P r 0 , π s and using the same logic as for (Hand & Voroninski, 2018, Eq. 17).
Revised: We establish (31) by using the fact that | g 1 p θ q| ď 1 for all θ P r 0 , π s and using the same logic as for (Hand & Voroninski, 2018, Eq. 17).

Original: We now establish (33).
Revised: We now establish (33).

Original: As θ 0 “ π ` O 1 p δ q , we have θ i “ q θ i ` O 1 p iδ q .
Revised: As θ 0 “ π ` O 1 p δ q , we have θ i “ q θ i ` O 1 p iδ q .

Original: Thus, if d 2 δπ ď 1 , d ´
Revised: Thus, if d 2 δπ ď 1 , d ´

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Thus (33) holds.
Revised: Thus (33) holds.

Original: Next, we establish that x P S β ñ r ď 4 d , and thus M ď 4 d .
Revised: Next, we establish that x P S β ñ r ď 4 d , and thus M ď 4 d .

Original: Suppose r ą 1 .
Revised: Suppose r ą 1 .

Original: At least one of the following holds: | sin θ 0 | ě 1 {? 2 or | cos θ 0 | ě 1 {? 2 .
Revised: At least one of the following holds: | sin θ 0 | ě 1 {? 2 or | cos θ 0 | ě 1 {? 2 .

Original: If | sin θ 0 | ě 1 {? 2 then (25) implies that | r ´ ζ | ď ? 2 βr .
Revised: If | sin θ 0 | ě 1 {? 2 then (25) implies that | r ´ ζ | ď ? 2 βr .

Original: Using (27), we get r ď d { π 1 ´?
Revised: Using (27), we get r ď d { π 1 ´?

Original: [Equation]
Revised: [Equation]

Original: ď d { 2 if β ă 1 { 4 .
Revised: ď d { 2 if β ă 1 { 4 .

Original: If | cos θ 0 | ě 1 {? 2 , then (24) implies that | r ´ ζ | ď ?
Revised: If | cos θ 0 | ě 1 {? 2 , then (24) implies that | r ´ ζ | ď ?

Original: 2 p βr ` | ξ |q .
Revised: 2 p βr ` | ξ |q .

Original: Using (26), (27), and β ă 1 { 4 , we get r ď ? 2 | ξ |` ζ 1 ´?
Revised: Using (26), (27), and β ă 1 { 4 , we get r ď ? 2 | ξ |` ζ 1 ´?

Original: 2 β ď d `? 2 1 ´?
Revised: 2 β ď d `? 2 1 ´?

Original: 2 β ď 4 d .
Revised: 2 β ď 4 d .

Original: Thus, we have x P S β ñ r ď 4 d ñ M ď 4 d .
Revised: Thus, we have x P S β ñ r ď 4 d ñ M ď 4 d .

Original: Next, we establish that we only need to consider the small angle case ( θ 0 « 0 ) and the large angle case ( θ 0 « π ), by considering the following three cases:
Revised: Next, we establish that we only need to consider the small angle case ( θ 0 « 0 ) and the large angle case ( θ 0 « π ), by considering the following three cases:

Original: (Case I) sin θ 0 ď 16 d 4 β : We have θ 

Revised: (Case I) sin θ 0 ď 16 d 4 β : We have θ

Original: “ O 1 p 32 d 4 β q or θ 0 “ π ` O 1 p 32 d 4 β q , as 32 d 4 β ă 1 .
Revised: “ O 1 p 32 d 4 β q or θ 0 “ π ` O 1 p 32 d 4 β q , as 32 d 4 β ă 1 .

Original: (Case II) | r ´ ζ | ă ? βM :
Revised: (Case II) | r ´ ζ | ă ? βM :

Original: Applying case II to inequality (24) yields | ξ | ď 2 ? βM .
Revised: Applying case II to inequality (24) yields | ξ | ď 2 ? βM .

Original: Using (30), we get θ 

Revised: Using (30), we get θ

Original: “ π ` O 1 p 2 πd 3 ? βM q .
Revised: “ π ` O 1 p 2 πd 3 ? βM q .

Original: (Case III) sin θ 0 ą 16 d 4 β and | r ´ ζ | ě ?
Revised: (Case III) sin θ 0 ą 16 d 4 β and | r ´ ζ | ě ?

Original: βM : Finally, consider Case III.
Revised: βM : Finally, consider Case III.

Original: By (25), we have | r ´ ζ | ď βM sin θ 0 .
Revised: By (25), we have | r ´ ζ | ď βM sin θ 0 .

Original: Using this inequality in (24), we have | ξ | ď βM ` βM sin θ 0 ď 2 βM sin θ 0 ď 18 d ´ 4 M
Revised: Using this inequality in (24), we have | ξ | ď βM ` βM sin θ 0 ď 2 βM sin θ 0 ď 18 d ´ 4 M

Original: ď 12 d ´ 3 , where the second to last inequality uses sin θ 0 ą 16 d 4 β and
Revised: ď 12 d ´ 3 , where the second to last inequality uses sin θ 0 ą 16 d 4 β and

Original: the last inequality uses M ď 4 d .
Revised: the last inequality uses M ď 4 d .

Original: By (30), we have π ´ θ 

Revised: By (30), we have π ´ θ

Original: π d ´ 3 ď ξ ď 12 d ´ 3 , which implies that θ 0 ě π { 2 .
Revised: π d ´ 3 ď ξ ď 12 d ´ 3 , which implies that θ 0 ě π { 2 .

Original: Now, as | r ´ ζ | ě ? βM , then by (25), we have | sin θ 0 | ď ?
Revised: Now, as | r ´ ζ | ě ? βM , then by (25), we have | sin θ 0 | ď ?

Original: β .
Revised: β .

Original: Hence, θ 

Revised: Hence, θ

Original: “ π ` O 1 p 2 ?
Revised: “ π ` O 1 p 2 ?

Original: β q , as θ 0 ě π { 2 and as β ă 1 .
Revised: β q , as θ 0 ě π { 2 and as β ă 1 .

Original: At least one of the Cases I,II, or III hold.
Revised: At least one of the Cases I,II, or III hold.

Original: Thus, we see that it sufﬁces to consider the small angle case θ 

Revised: Thus, we see that it sufﬁces to consider the small angle case θ

Original: “ O 1 p 32 d 4 β q or the large angle case θ 0 “ π ` O 1 p 8 πd 4 ?
Revised: “ O 1 p 32 d 4 β q or the large angle case θ 0 “ π ` O 1 p 8 πd 4 ?

Original: β q .
Revised: β q .

Original: Small Angle Case .
Revised: Small Angle Case .

Original: Assume θ 

Revised: Assume θ

Original: “ O 1 p δ q with δ “ 32 d 4 β .
Revised: “ O 1 p δ q with δ “ 32 d 4 β .

Original: As θ i ď θ 0 ď δ for all i , we have 1 ě ξ ě p 1 ´ δ π q d “ 1 ` O 1 p 2 δd π q provided δd { π ď 1 { 2 (which holds by our choice δ “ 32 d 4 β by assumption 64 d 6 ?
Revised: As θ i ď θ 0 ď δ for all i , we have 1 ě ξ ě p 1 ´ δ π q d “ 1 ` O 1 p 2 δd π q provided δd { π ď 1 { 2 (which holds by our choice δ “ 32 d 4 β by assumption 64 d 6 ?

Original: β ď 1 ).
Revised: β ď 1 ).

Original: By (27), we also have ζ “ O 1 p dπ δ q .
Revised: By (27), we also have ζ “ O 1 p dπ δ q .

Original: By (24), we have
Revised: By (24), we have

Original: [Equation]
Revised: [Equation]

Original: Thus, as cos θ 0 “
Revised: Thus, as cos θ 0 “

Original: [Equation]
Revised: [Equation]

Original: ` O 1 p
Revised: ` O 1 p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: , and r ď M ď 4 d (shown above) provides,
Revised: , and r ď M ď 4 d (shown above) provides,

Original: [Equation]
Revised: [Equation]

Original: By plugging in that δ “ 32 d 4 β , we have that r ´ 1 “ O 1 p 132 d 6 β q , where we have used that 32 d 5 β π ď 1 { 2 .
Revised: By plugging in that δ “ 32 d 4 β , we have that r ´ 1 “ O 1 p 132 d 6 β q , where we have used that 32 d 5 β π ď 1 { 2 .

Original: Large Angle Case .
Revised: Large Angle Case .

Original: Assume θ 

Revised: Assume θ

Original: “ π ` O 1 p δ q where δ “ 8 πd 4 ?
Revised: “ π ` O 1 p δ q where δ “ 8 πd 4 ?

Original: β .
Revised: β .

Original: By (32) and (33), we have ξ “ O 1 p δ { π q , and we have ζ “ ρ d ` O 1 p 3 d 3 δ q if 8 d 6 ?
Revised: By (32) and (33), we have ξ “ O 1 p δ { π q , and we have ζ “ ρ d ` O 1 p 3 d 3 δ q if 8 d 6 ?

Original: β ď 1 .
Revised: β ď 1 .

Original: By (24), we have
Revised: By (24), we have

Original: [Equation]
Revised: [Equation]

Original: , so, as cos θ 0 “ 1 ´
Revised: , so, as cos θ 0 “ 1 ´

Original: O 1 p θ 20 { 2 q ,
Revised: O 1 p θ 20 { 2 q ,

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: qq “ O 1 p βM q , and thus, using r ď 4 d , ρ d ď d , and δ “ 8 πd 4 ?
Revised: qq “ O 1 p βM q , and thus, using r ď 4 d , ρ d ď d , and δ “ 8 πd 4 ?

Original: β ď 1 ,
Revised: β ď 1 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To conclude the proof of (23), we use the fact that
Revised: To conclude the proof of (23), we use the fact that

Original: [Equation]
Revised: [Equation]

Original: This fact simply says that if a 2d point is known to have magnitude within ∆ r of some r and is known to be within angle ∆ θ from 0 , then its Euclidean distance to the point of polar coordinates p r, 0 q is no more than ∆ r ` p r ` ∆ r q ∆ θ .
Revised: This fact simply says that if a 2d point is known to have magnitude within ∆ r of some r and is known to be within angle ∆ θ from 0 , then its Euclidean distance to the point of polar coordinates p r, 0 q is no more than ∆ r ` p r ` ∆ r q ∆ θ .

Original: Finally, we establish that ρ d Ñ 1 as d Ñ 8 .
Revised: Finally, we establish that ρ d Ñ 1 as d Ñ 8 .

Original: Note that ρ d ` 1 “
Revised: Note that ρ d ` 1 “

Original: [Equation]
Revised: [Equation]

Original: and ρ 0 “ 0 .
Revised: and ρ 0 “ 0 .

Original: It sufﬁces to show ˜
Revised: It sufﬁces to show ˜

Original: ρ
Revised: ρ

Original: d Ñ 0 , where ˜ ρ d : “ 1 ´ ρ d .
Revised: d Ñ 0 , where ˜ ρ d : “ 1 ´ ρ d .

Original: The following recurrence relation holds:
Revised: The following recurrence relation holds:

Original: ˜ ρ d “ p
Revised: ˜ ρ d “ p

Original: [Equation]
Revised: [Equation]

Original: , with ˜ ρ 0 “ 1 .
Revised: , with ˜ ρ 0 “ 1 .

Original: Using the recurrence formula (Hand & Voroninski, 2018, Eq.
Revised: Using the recurrence formula (Hand & Voroninski, 2018, Eq.

Original: (15)) and the fact that q θ 0 “ π , we get that
Revised: (15)) and the fact that q θ 0 “ π , we get that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: using (29), we have that
Revised: using (29), we have that

Original: [Equation]
Revised: [Equation]

Original: Using (28) and the fact that q θ i ´ 1 ´ sin q
Revised: Using (28) and the fact that q θ i ´ 1 ´ sin q

Original: θ
Revised: θ

Original: i ´ 1 ď q θ 3 i
Revised: i ´ 1 ď q θ 3 i

Original: [Equation]
Revised: [Equation]

Original: 1 { 6 , we have that ˜ ρ d ď ř di “
Revised: 1 { 6 , we have that ˜ ρ d ď ř di “

Original: [Table]
Revised: [Table]

Revised: q θ 3 i

Original: [Table]
Revised: [Table]

Original:  π ¨ i
Revised: 6 π ¨ i

Original: [Table]
Revised: [Table]

Original: Ñ 0 as d Ñ 8 .
Revised: Ñ 0 as d Ñ 8 .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Consider the function f
Revised: Consider the function f

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: and note that f p x q “ f η p x q ` } η } 2 .
Revised: and note that f p x q “ f η p x q ` } η } 2 .

Original: Consider x P B p φ d x ˚ , ϕ } x ˚ }q , for a ϕ that will be speciﬁed later.
Revised: Consider x P B p φ d x ˚ , ϕ } x ˚ }q , for a ϕ that will be speciﬁed later.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds on the event E noise , by Lemma 5, and the last inequality holds by our assumption on x .
Revised: where the second inequality holds on the event E noise , by Lemma 5, and the last inequality holds by our assumption on x .

Original: Thus, for x P B p φ d x ˚ , ϕ } x ˚ }q
Revised: Thus, for x P B p φ d x ˚ , ϕ } x ˚ }q

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (cid:15) ă ? (cid:15) , ρ d ď 1 , 4 (cid:15)d ă 1 , ϕ
Revised: where the last inequality follows from (cid:15) ă ? (cid:15) , ρ d ď 1 , 4 (cid:15)d ă 1 , ϕ

Original: ă 1 and assuming ϕ “ (cid:15) .
Revised: ă 1 and assuming ϕ “ (cid:15) .

Original: Similarly, we have that for any y P B p´ φ d x ˚ , ϕ } x ˚ }q
Revised: Similarly, we have that for any y P B p´ φ d x ˚ , ϕ } x ˚ }q

Original: [Equation]
Revised: [Equation]

Original: Using (cid:15) ă ? (cid:15) , ρ d ď 1 , 4 (cid:15)d ă 1 , ϕ
Revised: Using (cid:15) ă ? (cid:15) , ρ d ď 1 , 4 (cid:15)d ă 1 , ϕ

Original: ă 1 and assuming ϕ “ (cid:15) , the right side of (43) is smaller than the right side of (44) if
Revised: ă 1 and assuming ϕ “ (cid:15) , the right side of (43) is smaller than the right side of (44) if

Original: [Equation]
Revised: [Equation]

Original: We can establish that:
Revised: We can establish that:

Original: Lemma 13.
Revised: Lemma 13.

Original: For all d ě 2 , that
Revised: For all d ě 2 , that

Original: [Equation]
Revised: [Equation]

Original: Thus, it sufﬁces to have ϕ “ (cid:15) “ K 3 d 10 and 13 } η } 2 ď K 9 d 2 ď
Revised: Thus, it sufﬁces to have ϕ “ (cid:15) “ K 3 d 10 and 13 } η } 2 ď K 9 d 2 ď

Original: K 2 K 1 p d ` 2 q 2 for an appropriate universal constant K 9 , and for an appropriate universal constant K 3 .
Revised: K 2 K 1 p d ` 2 q 2 for an appropriate universal constant K 9 , and for an appropriate universal constant K 3 .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: where θ x,y “ = p x
Revised: where θ x,y “ = p x

Original: [Equation]
Revised: [Equation]

Original: y q .
Revised: y q .

Original: We recall the results (36)
Revised: We recall the results (36)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and (50) in (Hand & Voroninski, 2018):
Revised: and (50) in (Hand & Voroninski, 2018):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have for all 0 ď i ď d ´ 2 , d ´ 1 ź j “ i `ˆ
Revised: Therefore, we have for all 0 ď i ď d ´ 2 , d ´ 1 ź j “ i `ˆ

Original: [Equation]
Revised: [Equation]

Original: j “ i ` 1 6 j ` 3 ě e ´ ş d ´ 1 i 6 s ` 3 ds “ ˆ i `d `˙, where the second and the ﬁfth inequalities follow from (49) and (50) respectively.
Revised: j “ i ` 1 6 j ` 3 ě e ´ ş d ´ 1 i 6 s ` 3 ds “ ˆ i `d `˙, where the second and the ﬁfth inequalities follow from (49) and (50) respectively.

Original: Since π 3 {p 12 p i ` 1 q 3 q ď ˇ θ 3 i { 12 ď ˇ θ
Revised: Since π 3 {p 12 p i ` 1 q 3 q ď ˇ θ 3 i { 12 ď ˇ θ

Original: i
Revised: i

Original: ´ sin ˇ θ i ď
Revised: ´ sin ˇ θ i ď

Original: ˇ θ 3 i { 6 ď 27 π 3 {p 6 p i ` 3 q 3 q , we have that for all d ě
Revised: ˇ θ 3 i { 6 ď 27 π 3 {p 6 p i ` 3 q 3 q , we have that for all d ě

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ď
Revised: ď

Original: [Equation]
Revised: [Equation]

Original: `and 1 ´ ρ d ě ˆ p d ` 2 q ˙` d ´ 1 ÿ i “
Revised: `and 1 ´ ρ d ě ˆ p d ` 2 q ˙` d ´ 1 ÿ i “

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ě
Revised: ě

Original: K 1 p
Revised: K 1 p

Original: [Equation]
Revised: [Equation]

Original: 2 q 2 , where we use ř 8 i “ 4 1 i 2 ď π 2 6 and ř ni “ 1 i 3 “ O p n 4 q .
Revised: 2 q 2 , where we use ř 8 i “ 4 1 i 2 ď π 2 6 and ř ni “ 1 i 3 “ O p n 4 q .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: To establish Lemma 11, we prove the following:
Revised: To establish Lemma 11, we prove the following:

Original: Lemma 14.
Revised: Lemma 14.

Original: For all x, y ‰ 0 ,
Revised: For all x, y ‰ 0 ,

Original: [Equation]
Revised: [Equation]

Original: x ´
Revised: x ´

Original: [Equation]
Revised: [Equation]

Original: Lemma 11 follows by noting that if x, y R B p 0 , r } x ˚ }q , then } h x ´ h y } ď ´ 12 d ` 6 d ` 4 d 2 πr 2 d ¯ } x ´ y } .
Revised: Lemma 11 follows by noting that if x, y R B p 0 , r } x ˚ }q , then } h x ´ h y } ď ´ 12 d ` 6 d ` 4 d 2 πr 2 d ¯ } x ´ y } .

Original: Proof of Lemma 14.
Revised: Proof of Lemma 14.

Original: For brevity of notation, let ζ j,z “ ś d ´ 1 i “ j π
Revised: For brevity of notation, let ζ j,z “ ś d ´ 1 i “ j π

Original: ´ ¯ θ i,z π .
Revised: ´ ¯ θ i,z π .

Original: Combining (46) and (47) gives | ¯ θ 0 ,x
Revised: Combining (46) and (47) gives | ¯ θ 0 ,x

Original: ´ ¯ θ 0 ,y | ď 4 max
Revised: ´ ¯ θ 0 ,y | ď 4 max

Original: [Equation]
Revised: [Equation]

Original: 1 } y } ¯ } x ´ y } .
Revised: 1 } y } ¯ } x ´ y } .

Original: Inequality (48) implies | ¯ θ i,x ´ ¯ θ i,y | ď | ¯ θ j,x ´ ¯ θ j,y | for all i ě j .
Revised: Inequality (48) implies | ¯ θ i,x ´ ¯ θ i,y | ď | ¯ θ j,x ´ ¯ θ j,y | for all i ě j .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: By Lemma 15, we have
Revised: By Lemma 15, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Additionally, it holds that
Revised: Additionally, it holds that

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using (46) and (47) and noting } ˆ x ´ ˆ y } ď θ x,y yield
Revised: Using (46) and (47) and noting } ˆ x ´ ˆ y } ď θ x,y yield

Original: [Equation]
Revised: [Equation]

Original: } x ´ y } .
Revised: } x ´ y } .

Original: Finally, combining (51), (52), (53), (54) and (55) yields the result.
Revised: Finally, combining (51), (52), (53), (54) and (55) yields the result.

Original: Lemma 15.
Revised: Lemma 15.

Original: Suppose a i , b i P r 0 , π s for i “ 1 , . . .
Revised: Suppose a i , b i P r 0 , π s for i “ 1 , . . .

Original: , k , and | a i ´
Revised: , k , and | a i ´

Original: b i | ď | a j ´ b j | , @ i ě j .
Revised: b i | ď | a j ´ b j | , @ i ě j .

Original: Then it holds that
Revised: Then it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Prove by induction.
Revised: Prove by induction.

Original: It is easy to verify that the inequality holds if k “ 1 .
Revised: It is easy to verify that the inequality holds if k “ 1 .

Original: Suppose the inequality holds with k “ t ´ 1 .
Revised: Suppose the inequality holds with k “ t ´ 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]