Original: U NDERSTANDING
Revised: U NDERSTANDING

Original: T RAIN -V ALIDATION
Revised: T RAIN -V ALIDATION

Original: S PLIT
Revised: S PLIT

Original: IN
Revised: IN

Original: M ETA -L EARNING WITH N EURAL N ETWORKS
Revised: M ETA -L EARNING WITH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of meta-learning is to learn a good prior model from a collection of tasks such that the learned prior is able to adapt quickly to new tasks without accessing many data from the new tasks.
Revised: The goal of meta-learning is to learn a good prior model from a collection of tasks such that the learned prior is able to adapt quickly to new tasks without accessing many data from the new tasks.

Original: A common practice in meta-learning is to perform a train-validation split on each task, where the training set is used for adapting the model parameter to that speciﬁc task and the validation set is used for learning a prior model that is shared across all tasks.
Revised: A common practice in meta-learning is to perform a train-validation split on each task, where the training set is used for adapting the model parameter to that speciﬁc task and the validation set is used for learning a prior model that is shared across all tasks.

Original: Despite its success and popularity in multitask learning and few-shot learning, the understanding of the train-validation split is still limited, especially when the neural network models are used.
Revised: Despite its success and popularity in multitask learning and few-shot learning, the understanding of the train-validation split is still limited, especially when the neural network models are used.

Original: In this paper, we study the beneﬁt of train-validation split for classiﬁcation problems with neural network models trained by gradient descent.
Revised: In this paper, we study the beneﬁt of train-validation split for classiﬁcation problems with neural network models trained by gradient descent.

Original: For ﬁrst-order model-agnostic meta-learning (FOMAML), we prove that the train-validation split is necessary to learn a good prior model when the noise in the training sample is large, while the train-train method fails.
Revised: For ﬁrst-order model-agnostic meta-learning (FOMAML), we prove that the train-validation split is necessary to learn a good prior model when the noise in the training sample is large, while the train-train method fails.

Original: We validate our theory by conducting experiment on both synthetic and real datasets.
Revised: We validate our theory by conducting experiment on both synthetic and real datasets.

Original: To the best of our knowledge, this is the ﬁrst work towards the theoretical understanding of train-validation split in metalearning with neural networks.
Revised: To the best of our knowledge, this is the ﬁrst work towards the theoretical understanding of train-validation split in metalearning with neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, meta-learning has gained increasing popularity and been successfully applied to a wide range of problems including few-shot learning (Ren et al., 2018;
Revised: In recent years, meta-learning has gained increasing popularity and been successfully applied to a wide range of problems including few-shot learning (Ren et al., 2018;

Original: Li et al., 2017; Rusu et al., 2018; Snell et al., 2017), reinforcement learning (Gupta et al., 2018b;a), neural machine translation (Gu et al., 2018), and neural architecture search (NAS) (Liu et al., 2018; Real et al., 2019).
Revised: Li et al., 2017; Rusu et al., 2018; Snell et al., 2017), reinforcement learning (Gupta et al., 2018b;a), neural machine translation (Gu et al., 2018), and neural architecture search (NAS) (Liu et al., 2018; Real et al., 2019).

Original: A popular meta-learning idea is to formulate it as a bi-level optimization problem, where the inner level computes the parameter adaptation to each task, while the outer level tries to minimize the meta-training loss.
Revised: A popular meta-learning idea is to formulate it as a bi-level optimization problem, where the inner level computes the parameter adaptation to each task, while the outer level tries to minimize the meta-training loss.

Original: Such a bi-level optimization formulation is empirically proved effective to learn new tasks quickly using only a few examples with the aid of past experience.
Revised: Such a bi-level optimization formulation is empirically proved effective to learn new tasks quickly using only a few examples with the aid of past experience.

Original: Following this idea, meta-learning algorithms such as model agnostic meta-learning (MAML)
Revised: Following this idea, meta-learning algorithms such as model agnostic meta-learning (MAML)

Original: (Finn et al., 2017) have achieved remarkable success in many applications.
Revised: (Finn et al., 2017) have achieved remarkable success in many applications.

Original: Due to the nature of bi-level optimization, meta-learning algorithms can often take advantage of a train-validation split in the dataset, so that the inner and outer levels of the algorithm use different data points (Finn et al., 2017; Rajeswaran et al., 2019; Bai et al., 2021;
Revised: Due to the nature of bi-level optimization, meta-learning algorithms can often take advantage of a train-validation split in the dataset, so that the inner and outer levels of the algorithm use different data points (Finn et al., 2017; Rajeswaran et al., 2019; Bai et al., 2021;

Original: Fallah et al., 2020).
Revised: Fallah et al., 2020).

Original: It is believed that the train-validation split can help the meta-learning algorithm to achieve a better performance.
Revised: It is believed that the train-validation split can help the meta-learning algorithm to achieve a better performance.

Original: There has been several attempts to understand the importance of train-validation split in meta-learning for linear models (Wang et al., 2021; Bai et al., 2021; Saunshi et al., 2021).
Revised: There has been several attempts to understand the importance of train-validation split in meta-learning for linear models (Wang et al., 2021; Bai et al., 2021; Saunshi et al., 2021).

Original: Speciﬁcally, Wang et al.
Revised: Speciﬁcally, Wang et al.

Original: (2021) showed that when learning linear models, the train-train method performs much worse than the train-validation method if the sample size is small and the noise is large.
Revised: (2021) showed that when learning linear models, the train-train method performs much worse than the train-validation method if the sample size is small and the noise is large.

Original: They also show that the train-train method is able to perform well on linear models when the sample size is large enough.
Revised: They also show that the train-train method is able to perform well on linear models when the sample size is large enough.

Original: Bai et al.
Revised: Bai et al.

Original: (2021) considered the linear centroid model introduced in Denevi et al.
Revised: (2021) considered the linear centroid model introduced in Denevi et al.

Original: (2018b)
Revised: (2018b)

Original: and showed that the train-validation method outperforms the train-train method in an agnostic setting, while in the realizable noiseless setting, the train-train method can asymptotically achieve a strictly smaller mean square error than the train-validation method as the sample size and dimension go to inﬁnity at a ﬁxed ratio.
Revised: and showed that the train-validation method outperforms the train-train method in an agnostic setting, while in the realizable noiseless setting, the train-train method can asymptotically achieve a strictly smaller mean square error than the train-validation method as the sample size and dimension go to inﬁnity at a ﬁxed ratio.

Original: Saunshi et al.
Revised: Saunshi et al.

Original: (2021) considered a representation learning perspective, and demonstrated that train-validation split encourages the learned representation to be low-rank while the train-train method encourages high-rank representations.
Revised: (2021) considered a representation learning perspective, and demonstrated that train-validation split encourages the learned representation to be low-rank while the train-train method encourages high-rank representations.

Original: However, all these works focus on the linear regression setting, while the advantage of train-validation split remains elusive for meta-learning with neural networks.
Revised: However, all these works focus on the linear regression setting, while the advantage of train-validation split remains elusive for meta-learning with neural networks.

Original: Based on the above observation, we raise the following question:
Revised: Based on the above observation, we raise the following question:

Original: How does train-validation split affect the meta learning with neural networks?
Revised: How does train-validation split affect the meta learning with neural networks?

Original: In this paper, we answer the above question via a case study of few-shot binary classiﬁcation using a two-layer convolutional neural network.
Revised: In this paper, we answer the above question via a case study of few-shot binary classiﬁcation using a two-layer convolutional neural network.

Original: We consider a learning problem where the data model consists of large noises, and only a limited number of data are available.
Revised: We consider a learning problem where the data model consists of large noises, and only a limited number of data are available.

Original: Under this setting, we theoretically compare the performance of ﬁrst-order MAML (FOMAML (Finn et al., 2017)), which is a simpliﬁcation of MAML that ignores the hessian terms, with a train-validation split (the trainvalidation method) and without a train-validation split (the train-train method).
Revised: Under this setting, we theoretically compare the performance of ﬁrst-order MAML (FOMAML (Finn et al., 2017)), which is a simpliﬁcation of MAML that ignores the hessian terms, with a train-validation split (the trainvalidation method) and without a train-validation split (the train-train method).

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: We show that under our setting, despite the complex bi-level structure of the FOMAML loss and its non-convex landscape, it is guaranteed that the train-validation method and the traintrain method can both train a two-layer CNN to a global minimum of the training loss with high probability.
Revised: We show that under our setting, despite the complex bi-level structure of the FOMAML loss and its non-convex landscape, it is guaranteed that the train-validation method and the traintrain method can both train a two-layer CNN to a global minimum of the training loss with high probability.

Original: We also demonstrate that there is a signiﬁcant performance gap on new test data.
Revised: We also demonstrate that there is a signiﬁcant performance gap on new test data.

Original: Speciﬁcally, we show that the neural network trained by the train-validation method can achieve a test loss that decreases exponentially fast as the number of training tasks increase.
Revised: Speciﬁcally, we show that the neural network trained by the train-validation method can achieve a test loss that decreases exponentially fast as the number of training tasks increase.

Original: On the other hand, we also show that the train-train method can at best achieve a constant level test loss.
Revised: On the other hand, we also show that the train-train method can at best achieve a constant level test loss.

Original: Our study demonstrates the importance of train-validation split in learning neural networks.
Revised: Our study demonstrates the importance of train-validation split in learning neural networks.

Original: To the best of our knowledge, this is the ﬁrst theoretical work studying train-validation split of metalearning with neural networks.
Revised: To the best of our knowledge, this is the ﬁrst theoretical work studying train-validation split of metalearning with neural networks.

Original: Notably, the learning problem we consider is linearly realizable, for which Bai et al.
Revised: Notably, the learning problem we consider is linearly realizable, for which Bai et al.

Original: (2021) showed that the train-train method can asymptotically achieve better MSE than the train-validation method under a linear model.
Revised: (2021) showed that the train-train method can asymptotically achieve better MSE than the train-validation method under a linear model.

Original: However, our results give the opposite conclusion for learning CNNs – the train-validation method still outperforms the train-train method even for linearly realizable learning problems.
Revised: However, our results give the opposite conclusion for learning CNNs – the train-validation method still outperforms the train-train method even for linearly realizable learning problems.

Original: Therefore, our results indicate that trainvalidation split may have a more signiﬁcant advantage when learning complicated prediction models.
Revised: Therefore, our results indicate that trainvalidation split may have a more signiﬁcant advantage when learning complicated prediction models.

Original: We perform experiments on both synthetic and real datasets with neural networks as backbone model to justify our theoretical results.
Revised: We perform experiments on both synthetic and real datasets with neural networks as backbone model to justify our theoretical results.

Original: In particular, even when the data and the neural network structure do not meet our theoretical assumptions, the experiment results still corroborate our theory to a certain extent.
Revised: In particular, even when the data and the neural network structure do not meet our theoretical assumptions, the experiment results still corroborate our theory to a certain extent.

Original: This demonstrates the practical value of our analysis.
Revised: This demonstrates the practical value of our analysis.

Original: Notation.
Revised: Notation.

Original: For an integer k , we denote [ k ] = { 1 , 2 , . . .
Revised: For an integer k , we denote [ k ] = { 1 , 2 , . . .

Original: , k } .
Revised: , k } .

Original: Given two sequences { x n } and { y n } with y n > 0 , we denote x n = O ( y n ) if | x n | /y n is upper bounded by a constant for all n .
Revised: Given two sequences { x n } and { y n } with y n > 0 , we denote x n = O ( y n ) if | x n | /y n is upper bounded by a constant for all n .

Original: Similarly, we denote x n = Ω( y n ) if | x n | /y n is lower bounded by a positive constant.
Revised: Similarly, we denote x n = Ω( y n ) if | x n | /y n is lower bounded by a positive constant.

Original: We denote x n = Θ( y n ) if x n = O ( y n ) and x n = Ω( y n ) .
Revised: We denote x n = Θ( y n ) if x n = O ( y n ) and x n = Ω( y n ) .

Original: Finally, we use (cid:101) O ( · ) , (cid:101) Ω( · ) , and (cid:101) Θ( · ) to hide logarithmic factors.
Revised: Finally, we use (cid:101) O ( · ) , (cid:101) Ω( · ) , and (cid:101) Θ( · ) to hide logarithmic factors.

Original:  A DDITIONAL R ELATED W ORK
Revised:  A DDITIONAL R ELATED W ORK

Original: Optimization and generalization guarantees for meta-learning.
Revised: Optimization and generalization guarantees for meta-learning.

Original: A number of recent works studied the optimization guarantees for meta-learning algorithms.
Revised: A number of recent works studied the optimization guarantees for meta-learning algorithms.

Original: Finn & Levine (2017) proved the universality of gradient based meta-learning.
Revised: Finn & Levine (2017) proved the universality of gradient based meta-learning.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) studied the global optimality conditions for MAML with a nonconvex objective.
Revised: (2020) studied the global optimality conditions for MAML with a nonconvex objective.

Original: Fallah et al.
Revised: Fallah et al.

Original: (2020) studied the convergence guarantee for MAML with nonconvex loss function and proposed Hessian-Free MAML with the same theoretical guarantee of MAML without accessing second order information.
Revised: (2020) studied the convergence guarantee for MAML with nonconvex loss function and proposed Hessian-Free MAML with the same theoretical guarantee of MAML without accessing second order information.

Original: Finn et al.
Revised: Finn et al.

Original: Balcan et al.
Revised: Balcan et al.

Original: ; Khodak et al.
Revised: ; Khodak et al.

Original: ; Denevi et al.
Revised: ; Denevi et al.

Original: (2019; 2018a) studied online meta-learning using online convex optimization.
Revised: (2019; 2018a) studied online meta-learning using online convex optimization.

Original: Another series of works have also studied the generalization error and sample complexities of meta-learning methods.
Revised: Another series of works have also studied the generalization error and sample complexities of meta-learning methods.

Original: Speciﬁcally, Amit & Meir (2018) extended the
Revised: Speciﬁcally, Amit & Meir (2018) extended the

Original: PAC-Bayes argument to the meta-learning setting and established a generalization error bound for meta-learning.
Revised: PAC-Bayes argument to the meta-learning setting and established a generalization error bound for meta-learning.

Original: Saunshi et al.
Revised: Saunshi et al.

Original: (2020) analyzed the sample complexity under the nonconvex setting using Reptile (Nichol et al., 2018).
Revised: (2020) analyzed the sample complexity under the nonconvex setting using Reptile (Nichol et al., 2018).

Original: There has also been a stream of work studying generalization in meta-learning from the representation learning perspective (Du et al., 2020; Tripuraneni et al., 2020; Denevi et al., 2018a).
Revised: There has also been a stream of work studying generalization in meta-learning from the representation learning perspective (Du et al., 2020; Tripuraneni et al., 2020; Denevi et al., 2018a).

Original: These works do not employ a train-validation split, instead they assume some underlying low-rank constraints on the representation.
Revised: These works do not employ a train-validation split, instead they assume some underlying low-rank constraints on the representation.

Original: Chen et al.
Revised: Chen et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2022) also studied generalization error bounds for meta-learning in the overparametrized regime.
Revised: (2022) also studied generalization error bounds for meta-learning in the overparametrized regime.

Original: Feature learning in over-parameterized neural networks.
Revised: Feature learning in over-parameterized neural networks.

Original: There is a series of recent work studying the feature learning dynamics of neural networks.
Revised: There is a series of recent work studying the feature learning dynamics of neural networks.

Original: For example, Allen-Zhu & Li (2020a) studied a sparse coding model and showed that adversarial training can help CNN ﬁlters to learn “pure” dictionary bases.
Revised: For example, Allen-Zhu & Li (2020a) studied a sparse coding model and showed that adversarial training can help CNN ﬁlters to learn “pure” dictionary bases.

Original: Allen-Zhu & Li (2020b) studied the impact of ensemble and knowledge distillation on the feature learning process.
Revised: Allen-Zhu & Li (2020b) studied the impact of ensemble and knowledge distillation on the feature learning process.

Original: Zou et al.
Revised: Zou et al.

Original: (2021) demonstrated the generalization gap between
Revised: (2021) demonstrated the generalization gap between

Original: Adam and stochastic gradient descent through the lens of feature learning.
Revised: Adam and stochastic gradient descent through the lens of feature learning.

Original: Cao et al.
Revised: Cao et al.

Original: (2022) studied feature learning and noise memorization in learning two-layer CNNs and showed a phase transition between benign and harmful overﬁtting.
Revised: (2022) studied feature learning and noise memorization in learning two-layer CNNs and showed a phase transition between benign and harmful overﬁtting.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: In this section, we introduce our data model, neural network model, loss function, and the details about the FOMAML algorithm with train-train and train-validation methods.
Revised: In this section, we introduce our data model, neural network model, loss function, and the details about the FOMAML algorithm with train-train and train-validation methods.

Original: We ﬁrst introduce our data model.
Revised: We ﬁrst introduce our data model.

Original: In meta-learning, the goal is to train a model based on the data from K tasks, so that the trained model can learn a new task efﬁciently.
Revised: In meta-learning, the goal is to train a model based on the data from K tasks, so that the trained model can learn a new task efﬁciently.

Original: Theoretical analysis of meta-learning thus requires careful modeling of (i) the relation among different tasks, and (ii) the data distribution for each speciﬁc task.
Revised: Theoretical analysis of meta-learning thus requires careful modeling of (i) the relation among different tasks, and (ii) the data distribution for each speciﬁc task.

Original: To achieve this, we suppose that the data distribution D k for the k -th task is deﬁned based on a vector ν k
Revised: To achieve this, we suppose that the data distribution D k for the k -th task is deﬁned based on a vector ν k

Original: , i.e., D k = D ( ν k ) , and that the vectors ν 1 , . .
Revised: , i.e., D k = D ( ν k ) , and that the vectors ν 1 , . .

Original: , ν K are independently drawn from a distribution Π .
Revised: , ν K are independently drawn from a distribution Π .

Original: Following this framework, the data points of a new task can be generated by ﬁrst sampling a vector ˜ ν from Π , then sampling data from the distribution
Revised: Following this framework, the data points of a new task can be generated by ﬁrst sampling a vector ˜ ν from Π , then sampling data from the distribution

Original: D (˜ ν ) .
Revised: D (˜ ν ) .

Original: In the following, we present the detailed deﬁnitions of Π and D (˜ ν ) .
Revised: In the following, we present the detailed deﬁnitions of Π and D (˜ ν ) .

Original: Deﬁnition 3.1 (Distribution of tasks) .
Revised: Deﬁnition 3.1 (Distribution of tasks) .

Original: Let ν , z 1 , . .
Revised: Let ν , z 1 , . .

Original: z M ∈ R d be ﬁxed vectors, where z 1 , . .
Revised: z M ∈ R d be ﬁxed vectors, where z 1 , . .

Original: z M are orthogonal to ν .
Revised: z M are orthogonal to ν .

Original: A vector ˜ ν is generated from Π by (i) randomly pick a vector z from { z 1 , . . .
Revised: A vector ˜ ν is generated from Π by (i) randomly pick a vector z from { z 1 , . . .

Original: , z M } , and (ii) let ˜ ν
Revised: , z M } , and (ii) let ˜ ν

Original: = ν
Revised: = ν

Original: + z .
Revised: + z .

Original: Deﬁnition 3.1 is based on a set of ﬁxed vectors ν , z 1 , . . .
Revised: Deﬁnition 3.1 is based on a set of ﬁxed vectors ν , z 1 , . . .

Original: , z M ∈ R d .
Revised: , z M ∈ R d .

Original: Here ν captures the feature shared by all tasks.
Revised: Here ν captures the feature shared by all tasks.

Original: And z 1 , . . . ,
Revised: And z 1 , . . . ,

Original: z M give a dictionary of possible unique features of each speciﬁc task.
Revised: z M give a dictionary of possible unique features of each speciﬁc task.

Original: It is important to note that we focus on the setting where the number of observed tasks K (cid:28) M .
Revised: It is important to note that we focus on the setting where the number of observed tasks K (cid:28) M .

Original: Under our setting, with high probability, different tasks will use different unique features.
Revised: Under our setting, with high probability, different tasks will use different unique features.

Original: Deﬁnition 3.2 (Distribution of data) .
Revised: Deﬁnition 3.2 (Distribution of data) .

Original: Given a vector ˜ ν ∈ R d , each data point ( x , y ) with x = [ x (1) (cid:62) , x (2) (cid:62) ] (cid:62) ∈
Revised: Given a vector ˜ ν ∈ R d , each data point ( x , y ) with x = [ x (1) (cid:62) , x (2) (cid:62) ] (cid:62) ∈

Original: R 2 d and y ∈ {− 1 , 1 } is generated from D (˜ ν ) as follows:
Revised: R 2 d and y ∈ {− 1 , 1 } is generated from D (˜ ν ) as follows:

Original: The label y is assigned as +1 or − 1 with equal probability.
Revised: The label y is assigned as +1 or − 1 with equal probability.

Original: A noise vector ξ is generated from N ( 0 , σ 2 ξ · ( I − P )) , where P ∈ R d × d is the projection operator onto span( { ν , z 1 , . .
Revised: A noise vector ξ is generated from N ( 0 , σ 2 ξ · ( I − P )) , where P ∈ R d × d is the projection operator onto span( { ν , z 1 , . .

Original: . , z M } ) .
Revised: . , z M } ) .

Original: One of x (1) , x (2) is randomly selected and assigned as y · ν k ; the other is assigned as ξ .
Revised: One of x (1) , x (2) is randomly selected and assigned as y · ν k ; the other is assigned as ξ .

Original: For k ∈ [ K ] , we denote by S k
Revised: For k ∈ [ K ] , we denote by S k

Original: = { ( x k,i , y k,i ) } ni =1 the set of independent samples from the kth observed task.
Revised: = { ( x k,i , y k,i ) } ni =1 the set of independent samples from the kth observed task.

Original: We consider a speciﬁc type of data input that consists of two patches, x k,i =
Revised: We consider a speciﬁc type of data input that consists of two patches, x k,i =

Original: [ x (1) (cid:62) k,i , x (2) (cid:62) k,i ] (cid:62) , to meet our study of convolutional neural networks.
Revised: [ x (1) (cid:62) k,i , x (2) (cid:62) k,i ] (cid:62) , to meet our study of convolutional neural networks.

Original: Note also that Deﬁnition 3. requires the noises in the data input to be sampled from the Gaussian distribution N ( 0 , σ 2 ξ · ( I − P )) to ensure that the noise vector ξ is orthogonal to the features ν , z 1 , . .
Revised: Note also that Deﬁnition 3. requires the noises in the data input to be sampled from the Gaussian distribution N ( 0 , σ 2 ξ · ( I − P )) to ensure that the noise vector ξ is orthogonal to the features ν , z 1 , . .

Original: . , z M .
Revised: . , z M .

Original: It is then easy to check that our data model is linearly realizable : the linear predictor θ ∗ =
Revised: It is then easy to check that our data model is linearly realizable : the linear predictor θ ∗ =

Original: (cid:107) ν (cid:107) − 2 2 · [ ν (cid:62) , ν (cid:62) ] (cid:62) ∈ R 2 d satisfy (cid:104) θ ∗ , x (cid:105) =
Revised: (cid:107) ν (cid:107) − 2 2 · [ ν (cid:62) , ν (cid:62) ] (cid:62) ∈ R 2 d satisfy (cid:104) θ ∗ , x (cid:105) =

Original: y for all ( x , y ) drawn from our data distribution.
Revised: y for all ( x , y ) drawn from our data distribution.

Original: Our motivation to study this linearly realizable setting is that it has been proved in Bai et al.
Revised: Our motivation to study this linearly realizable setting is that it has been proved in Bai et al.

Original: (2021) that the train-train method is strictly better than the train-validation method when learning linear models in the realizable setting.
Revised: (2021) that the train-train method is strictly better than the train-validation method when learning linear models in the realizable setting.

Original: On the contrary, in this paper we aim to show that for the linearly realizable data model deﬁned in
Revised: On the contrary, in this paper we aim to show that for the linearly realizable data model deﬁned in

Original: Deﬁnitions 3.1 and 3.2, the train-validation method can still outperform the train-train method for
Revised: Deﬁnitions 3.1 and 3.2, the train-validation method can still outperform the train-train method for

Original: FOMAML in learning two-layer CNNs.
Revised: FOMAML in learning two-layer CNNs.

Original: We study a two-layer CNN with m hidden layer neurons whose second layer weights are frozen as ± 1 ’s. We use the Huberized- ReLU (Chatterji et al., 2021) activation function, which is deﬁned as
Revised: We study a two-layer CNN with m hidden layer neurons whose second layer weights are frozen as ± 1 ’s. We use the Huberized- ReLU (Chatterji et al., 2021) activation function, which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where we set h = 1 / 2 in our analysis.
Revised: where we set h = 1 / 2 in our analysis.

Original: Let W represent the collection of all weights of our network.
Revised: Let W represent the collection of all weights of our network.

Original: For a data input x = [ x (1) (cid:62) , x (2) (cid:62) ] (cid:62) , we consider the CNN f ( W , x ) =
Revised: For a data input x = [ x (1) (cid:62) , x (2) (cid:62) ] (cid:62) , we consider the CNN f ( W , x ) =

Original: F +1 ( W +1 , x ) −
Revised: F +1 ( W +1 , x ) −

Original: F − 1 ( W − 1 , x ) , where
Revised: F − 1 ( W − 1 , x ) , where

Original: [Equation]
Revised: [Equation]

Original: We use w j,r to denote the r -th convolution ﬁlter with second layer weight j , and use W j to denote the collection of w j, 1 , . . .
Revised: We use w j,r to denote the r -th convolution ﬁlter with second layer weight j , and use W j to denote the collection of w j, 1 , . . .

Original: , w j,m .
Revised: , w j,m .

Original: We consider cross-entropy loss.
Revised: We consider cross-entropy loss.

Original: The loss of a data point ( x , y ) is given by L ( W , x , y ) =
Revised: The loss of a data point ( x , y ) is given by L ( W , x , y ) =

Original: (cid:96) [ y · f ( W , x )] , where (cid:96) ( z ) =
Revised: (cid:96) [ y · f ( W , x )] , where (cid:96) ( z ) =

Original: log(1 + exp( − z )) .
Revised: log(1 + exp( − z )) .

Original: The loss of a collection of data points S is deﬁned by
Revised: The loss of a collection of data points S is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Following the data model given in Deﬁnitions 3.1 and 3.2, we deﬁne the test loss achieved by a
Revised: Following the data model given in Deﬁnitions 3.1 and 3.2, we deﬁne the test loss achieved by a

Original: CNN with weights W as L test (
Revised: CNN with weights W as L test (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We implement the FOMAML algorithm
Revised: We implement the FOMAML algorithm

Original: (Finn et al., 2017) to train the neural network.
Revised: (Finn et al., 2017) to train the neural network.

Original: The train-train and train-validation methods are given as follows.
Revised: The train-train and train-validation methods are given as follows.

Original: Train-train : for each task k , we use all of the samples for adapting the parameter in the inner-loop updates.
Revised: Train-train : for each task k , we use all of the samples for adapting the parameter in the inner-loop updates.

Original: Speciﬁcally, the meta objective is to minimize
Revised: Speciﬁcally, the meta objective is to minimize

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) W ( W , S k ) represents the weights of the network after J gradient descent steps (w.r.t.
Revised: where (cid:101) W ( W , S k ) represents the weights of the network after J gradient descent steps (w.r.t.

Original: loss
Revised: loss

Original: L ( · , S k ) ) starting from W with step size γ .
Revised: L ( · , S k ) ) starting from W with step size γ .

Original: The FOMAML algorithm updates the CNN weights using the following gradient update rule with step size η :
Revised: The FOMAML algorithm updates the CNN weights using the following gradient update rule with step size η :

Original: [Equation]
Revised: [Equation]

Original: Train-validation : for each task k , we denote by I tr k
Revised: Train-validation : for each task k , we denote by I tr k

Original: = { 1 , . . . , n 1 } the training data indices, and
Revised: = { 1 , . . . , n 1 } the training data indices, and

Original: I val k
Revised: I val k

Original: = { n 1 + 1 , . . . , n } the validation data indices.
Revised: = { n 1 + 1 , . . . , n } the validation data indices.

Original: We then use S tr k = { ( x k,i , y k,i ) } i ∈I tr k as the training data set, and S val k
Revised: We then use S tr k = { ( x k,i , y k,i ) } i ∈I tr k as the training data set, and S val k

Original: = { ( x k,i , y k,i ) } i ∈I val k as the validation data set.
Revised: = { ( x k,i , y k,i ) } i ∈I val k as the validation data set.

Original: The meta objective of the train-validation method is to minimize
Revised: The meta objective of the train-validation method is to minimize

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) W ( W , S tr k ) represents the weights of the network after J gradient descent steps (w.r.t.
Revised: where (cid:101) W ( W , S tr k ) represents the weights of the network after J gradient descent steps (w.r.t.

Original: loss
Revised: loss

Original: L ( · , S tr k ) ) starting from W with step size γ .
Revised: L ( · , S tr k ) ) starting from W with step size γ .

Original: For the train-validation method, the FOMAML algorithm implements the following outer-loop update rule to train the network:
Revised: For the train-validation method, the FOMAML algorithm implements the following outer-loop update rule to train the network:

Original: [Equation]
Revised: [Equation]

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: In this section, we present our theoretical results for the train-train and train-validation methods.
Revised: In this section, we present our theoretical results for the train-train and train-validation methods.

Original: We ﬁrst introduce the following condition.
Revised: We ﬁrst introduce the following condition.

Original: Condition 4.1.
Revised: Condition 4.1.

Original: There exists σ s > 0 such that (1 / 2) · σ s √ d ≤ (cid:107) z
Revised: There exists σ s > 0 such that (1 / 2) · σ s √ d ≤ (cid:107) z

Original: i (cid:107) 2 ≤ (3 / 2) · σ s √ d for i ∈ [ M ] , and (cid:104) z i , z j (cid:105) ≤ O ( σ 2 s · (cid:112) d log( d )) for all i (cid:54) = j .
Revised: i (cid:107) 2 ≤ (3 / 2) · σ s √ d for i ∈ [ M ] , and (cid:104) z i , z j (cid:105) ≤ O ( σ 2 s · (cid:112) d log( d )) for all i (cid:54) = j .

Original: Condition 4.1 speciﬁes the properties of the task-speciﬁc features in Deﬁnition 3.1.
Revised: Condition 4.1 speciﬁes the properties of the task-speciﬁc features in Deﬁnition 3.1.

Original: Here we require that the task-speciﬁc features to have weak correlations to each other, which is standard in dictionary learning.
Revised: Here we require that the task-speciﬁc features to have weak correlations to each other, which is standard in dictionary learning.

Original: Moreover, it is easy to see that such z 1 , . . .
Revised: Moreover, it is easy to see that such z 1 , . . .

Original: , z M exist as long as M =
Revised: , z M exist as long as M =

Original: O (poly( d )) : with a simple Gaussian concentration argument, one can show that M independent zero-mean Gaussian random vectors with covariance matrix σ 2 s ( I − νν (cid:62) / (cid:107)
Revised: O (poly( d )) : with a simple Gaussian concentration argument, one can show that M independent zero-mean Gaussian random vectors with covariance matrix σ 2 s ( I − νν (cid:62) / (cid:107)

Original: ν (cid:107) 22 ) are orthogonal to ν and satisfy Condition 4.1 with high probability.
Revised: ν (cid:107) 22 ) are orthogonal to ν and satisfy Condition 4.1 with high probability.

Original: With Condition 4.1 and our data model in Deﬁnitions 3.1 and 3.2, it is necessary for a neural network to learn the shared feature ν to make accurate predictions on a new task.
Revised: With Condition 4.1 and our data model in Deﬁnitions 3.1 and 3.2, it is necessary for a neural network to learn the shared feature ν to make accurate predictions on a new task.

Original: On the other hand, if a neural network only utilizes the task-speciﬁc features or the noises in the observed data to ﬁt the labels, then it can only achieve a good training loss, but will have a poor prediction accuracy on new tasks.
Revised: On the other hand, if a neural network only utilizes the task-speciﬁc features or the noises in the observed data to ﬁt the labels, then it can only achieve a good training loss, but will have a poor prediction accuracy on new tasks.

Original: Following this intuition, we aim to construct a concrete setting under which the train-train and train-validation methods give different test losses while both achieving small training losses.
Revised: Following this intuition, we aim to construct a concrete setting under which the train-train and train-validation methods give different test losses while both achieving small training losses.

Original: The details of our constructed setting is summarized in the following condition.
Revised: The details of our constructed setting is summarized in the following condition.

Original: [Equation]
Revised: [Equation]

Original: 2 ) ≤ M ≤ d/ 2 .
Revised: 2 ) ≤ M ≤ d/ 2 .

Original: Condition 4.2 deﬁnes an over-parameterized setting where d (cid:29) Kn .
Revised: Condition 4.2 deﬁnes an over-parameterized setting where d (cid:29) Kn .

Original: Moreover, under Condition 4.2, the norms of the shared and task-speciﬁc features (cid:107) ν (cid:107) 2 , (cid:107) z 1 (cid:107) 2 , . . .
Revised: Moreover, under Condition 4.2, the norms of the shared and task-speciﬁc features (cid:107) ν (cid:107) 2 , (cid:107) z 1 (cid:107) 2 , . . .

Original: , (cid:107) z K (cid:107) 2 are all O (1) .
Revised: , (cid:107) z K (cid:107) 2 are all O (1) .

Original: In comparison, with high probability, the norms of noise vectors in the data are Θ(polylog( d )) .
Revised: In comparison, with high probability, the norms of noise vectors in the data are Θ(polylog( d )) .

Original: Therefore, Condition 4.2 deﬁnes a setting with relatively large noises.
Revised: Therefore, Condition 4.2 deﬁnes a setting with relatively large noises.

Original: The FOMAML algorithm involves inner and outer-loop training of the neural network.
Revised: The FOMAML algorithm involves inner and outer-loop training of the neural network.

Original: We specify our detailed hyper-parameter conﬁgurations in the following condition.
Revised: We specify our detailed hyper-parameter conﬁgurations in the following condition.

Original: Condition 4.3.
Revised: Condition 4.3.

Original: In the FOMAML algorithm deﬁned in Eq.
Revised: In the FOMAML algorithm deﬁned in Eq.

Original: (3.1) and Eq. (3.2), we initialize the
Revised: (3.1) and Eq. (3.2), we initialize the

Original: CNN weights W (0) by Gaussian random initialization with standard deviation σ 0 =
Revised: CNN weights W (0) by Gaussian random initialization with standard deviation σ 0 =

Original: d − 1 / 2 .
Revised: d − 1 / 2 .

Original: We set the inner-loop step size γ = polylog( d ) , and the outer-loop step size η = 1 / polylog( d ) .
Revised: We set the inner-loop step size γ = polylog( d ) , and the outer-loop step size η = 1 / polylog( d ) .

Original: We run T = poly( d ) outer-loop iterations.
Revised: We run T = poly( d ) outer-loop iterations.

Original: Within each outer-loop iteration, we run J = 5 inner-loop gradient descent steps.
Revised: Within each outer-loop iteration, we run J = 5 inner-loop gradient descent steps.

Original: In Condition 4.3, we use a slightly larger step size for inner-loop compared to that for the outer-loop.
Revised: In Condition 4.3, we use a slightly larger step size for inner-loop compared to that for the outer-loop.

Original: This is to ensure that the inner-loop updates can make a difference, and it matches the meta-learning practice.
Revised: This is to ensure that the inner-loop updates can make a difference, and it matches the meta-learning practice.

Original: Moreover, we set J = 5 to demonstrate that our analysis applies to the case with multiple inner-loop iterations.
Revised: Moreover, we set J = 5 to demonstrate that our analysis applies to the case with multiple inner-loop iterations.

Original: The exact value of J is not essential – we can easily apply our analysis to other values of J = Θ(1) .
Revised: The exact value of J is not essential – we can easily apply our analysis to other values of J = Θ(1) .

Original: We are now in the position to state our main theoretical results.
Revised: We are now in the position to state our main theoretical results.

Original: The theorem below gives the training and test loss guarantees for the train-train method.
Revised: The theorem below gives the training and test loss guarantees for the train-train method.

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: Under Conditions 4.1, 4.2 and 4.3, suppose that one uses the train-train method to train the neural network.
Revised: Under Conditions 4.1, 4.2 and 4.3, suppose that one uses the train-train method to train the neural network.

Original: Then with probability at least 1 − ( Kn ) − 10 , 1.
Revised: Then with probability at least 1 − ( Kn ) − 10 , 1.

Original: the
Revised: the

Original: training loss is small: min t ∈ [ T ] (cid:98) L tr − tr ( W ( t ) , {S k } Kk =1 ) ≤ O (cid:18) poly( d ) (cid:19) .
Revised: training loss is small: min t ∈ [ T ] (cid:98) L tr − tr ( W ( t ) , {S k } Kk =1 ) ≤ O (cid:18) poly( d ) (cid:19) .

Original: the test loss is large: min t ∈ [ T ] L test ( W ( t ) )
Revised: the test loss is large: min t ∈ [ T ] L test ( W ( t ) )

Original: = Ω(1) .
Revised: = Ω(1) .

Original: Theorem 4.4 demonstrates that FOMAML with train-train method can successfully train the CNN to minimize the training loss.
Revised: Theorem 4.4 demonstrates that FOMAML with train-train method can successfully train the CNN to minimize the training loss.

Original: However, it also shows that the train-train method fails on new tasks and can only achieve a constant level test loss.
Revised: However, it also shows that the train-train method fails on new tasks and can only achieve a constant level test loss.

Original: In comparison, we have the following theorem for the train-validation method.
Revised: In comparison, we have the following theorem for the train-validation method.

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: Under Conditions 4.1, 4.2 and 4.3, suppose that one uses the train-validation method to train the neural network.
Revised: Under Conditions 4.1, 4.2 and 4.3, suppose that one uses the train-validation method to train the neural network.

Original: Then with probability at least 1 − ( Kn ) − 10 ,
Revised: Then with probability at least 1 − ( Kn ) − 10 ,

Original: the training loss is small:
Revised: the training loss is small:

Original: [Equation]
Revised: [Equation]

Original: the test loss is also small: there exists a constant c > 0 such that
Revised: the test loss is also small: there exists a constant c > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Although both methods can achieve a 1 / poly( d ) training loss, the train-validation method is able to achieve a test loss of O (exp( − K c )) compared to a constant test loss Ω(1) by the train-train method.
Revised: Although both methods can achieve a 1 / poly( d ) training loss, the train-validation method is able to achieve a test loss of O (exp( − K c )) compared to a constant test loss Ω(1) by the train-train method.

Original: Thus, Theorems 4.4 and 4.5 show that for our data model, it is necessary to perform a train-validation split to achieve good performance on test data.
Revised: Thus, Theorems 4.4 and 4.5 show that for our data model, it is necessary to perform a train-validation split to achieve good performance on test data.

Original: We also compare FOMAML with the vanilla supervised learning framework where one simply combines all data from the K tasks together and uses gradient descent to minimize the overall crossentropy loss.
Revised: We also compare FOMAML with the vanilla supervised learning framework where one simply combines all data from the K tasks together and uses gradient descent to minimize the overall crossentropy loss.

Original: In fact, the results of Theorem 4.4 still hold even for J = 0 , where algorithm Eq.
Revised: In fact, the results of Theorem 4.4 still hold even for J = 0 , where algorithm Eq.

Original: (3.1) reduces to gradient descent for vanilla supervised learning.
Revised: (3.1) reduces to gradient descent for vanilla supervised learning.

Original: Therefore, vanilla supervised learning can also only achieve Θ(1) test loss.
Revised: Therefore, vanilla supervised learning can also only achieve Θ(1) test loss.

Original: Clearly, our results imply that FOMAML with train-validation split can signiﬁcantly outperform vanilla supervised learning when learning our data model.
Revised: Clearly, our results imply that FOMAML with train-validation split can signiﬁcantly outperform vanilla supervised learning when learning our data model.

Original: Comparison with Bai et al.
Revised: Comparison with Bai et al.

Original: Recently, Bai et al.
Revised: Recently, Bai et al.

Original: (2021) studied the importance of the trainvalidation split in meta-learning under the linear centroid framework.
Revised: (2021) studied the importance of the trainvalidation split in meta-learning under the linear centroid framework.

Original: Speciﬁcally, they considered linear ridge regression in the inner-loop and linear ridgeless regression in the outer-loop, and showed that the train-train method strictly outperforms the train-validation method in the realizable setting.
Revised: Speciﬁcally, they considered linear ridge regression in the inner-loop and linear ridgeless regression in the outer-loop, and showed that the train-train method strictly outperforms the train-validation method in the realizable setting.

Original: As we have discussed in Section 3, our data model is linearly realizable, and thus by Bai et al.
Revised: As we have discussed in Section 3, our data model is linearly realizable, and thus by Bai et al.

Original: (2021), the train-train method should outperform the train-validation method when learning our data model with linear ridge/ridgeless regression.
Revised: (2021), the train-train method should outperform the train-validation method when learning our data model with linear ridge/ridgeless regression.

Original: On the contrary, Theorems 4.4 and 4.5 demonstrate that the train-validation method still signiﬁcantly outperforms the train-train method for FOMAML in learning CNNs.
Revised: On the contrary, Theorems 4.4 and 4.5 demonstrate that the train-validation method still signiﬁcantly outperforms the train-train method for FOMAML in learning CNNs.

Original: We believe the key reason behind this difference is that the CNN model has a much higher expressive power, and therefore can more easily overﬁt noises in the training data points when using FOMAML.
Revised: We believe the key reason behind this difference is that the CNN model has a much higher expressive power, and therefore can more easily overﬁt noises in the training data points when using FOMAML.

Original: The train-validation split thus greatly helps feature learning of the neural network by performing a sample splitting between the inner and outer-loops.
Revised: The train-validation split thus greatly helps feature learning of the neural network by performing a sample splitting between the inner and outer-loops.

Original: Therefore, our results indicate that train-validation split may have a more signiﬁcant advantage when learning complicated prediction models.
Revised: Therefore, our results indicate that train-validation split may have a more signiﬁcant advantage when learning complicated prediction models.

Original:  E XPERIMENT 5. E XPERIMENTAL S ETUPS
Revised:  E XPERIMENT 5. E XPERIMENTAL S ETUPS

Original: Synthetic data.
Revised: Synthetic data.

Original: We generate synthetic data to test our theory.
Revised: We generate synthetic data to test our theory.

Original: For our data generation we choose: d = 1000 , K = 343 , n = 10 , σ ξ = 10 . 42 , σ s = 0 . 00066 , (cid:107) ν (cid:107) 2 = 1 .
Revised: For our data generation we choose: d = 1000 , K = 343 , n = 10 , σ ξ = 10 . 42 , σ s = 0 . 00066 , (cid:107) ν (cid:107) 2 = 1 .

Original: For our neural network we choose: m = 18 , σ 0 = 0 .
Revised: For our neural network we choose: m = 18 , σ 0 = 0 .

Original: And ﬁnally we choose the following parameters for inner and outer level optimization: γ = 0 .
Revised: And ﬁnally we choose the following parameters for inner and outer level optimization: γ = 0 .

Original: 001 , J = 5 , η = 0 . 0001 .
Revised: 001 , J = 5 , η = 0 . 0001 .

Original: Real-world Data.
Revised: Real-world Data.

Original: In our experiments, we further justify our theoretical ﬁndings in two real-world datasets: RainbowMNIST, miniImagenet, which are discussed as follows
Revised: In our experiments, we further justify our theoretical ﬁndings in two real-world datasets: RainbowMNIST, miniImagenet, which are discussed as follows

Original: • RainbowMNIST.
Revised: • RainbowMNIST.

Original: Following (Yao et al., 2021), RainbowMNIST is a 10-way meta-learning dataset built upon original MNIST dataset, where each task is constructed by applying one combination of image transformations (e.g., coloring, rotation) on the original data.
Revised: Following (Yao et al., 2021), RainbowMNIST is a 10-way meta-learning dataset built upon original MNIST dataset, where each task is constructed by applying one combination of image transformations (e.g., coloring, rotation) on the original data.

Original: Here, 40 andcombinations are used for meta-training and meta-testing, respectively.
Revised: Here, 40 andcombinations are used for meta-training and meta-testing, respectively.

Original: • miniImagenet.
Revised: • miniImagenet.

Original: Following the traditional meta-learning setting (Finn & Levine, 2017; Snell et al., 2017), miniImagenet dataset is split into meta-training, meta-validation and meta-testing classes, where 64/16/20 classes are used for meta-training/validation/testing.
Revised: Following the traditional meta-learning setting (Finn & Levine, 2017; Snell et al., 2017), miniImagenet dataset is split into meta-training, meta-validation and meta-testing classes, where 64/16/20 classes are used for meta-training/validation/testing.

Original: We adopt the traditional Nway, K-shot setting to split the training and validation set in our experiment, where N=5 and K= in this paper (i.e., 5-way, 1-shot
Revised: We adopt the traditional Nway, K-shot setting to split the training and validation set in our experiment, where N=5 and K= in this paper (i.e., 5-way, 1-shot

Original: learning).
Revised: learning).

Original: Backbones and Hyperparameters.
Revised: Backbones and Hyperparameters.

Original: For all real-world datasets, follow (Finn & Levine, 2017; Snell et al., 2017), we adopt the standard four-block convolutional layers as the base learner, where
Revised: For all real-world datasets, follow (Finn & Levine, 2017; Snell et al., 2017), we adopt the standard four-block convolutional layers as the base learner, where

Original: Huberized-ReLU is used as the activation function.
Revised: Huberized-ReLU is used as the activation function.

Original: The number of inner-loop steps is set as 5.
Revised: The number of inner-loop steps is set as 5.

Original: The inner-loop and outer-loop learning rates are set as: 0.01 and 0.001 (miniImagenet), 0.1 and 0.
Revised: The inner-loop and outer-loop learning rates are set as: 0.01 and 0.001 (miniImagenet), 0.1 and 0.

Original: (RainbowMNIST), respectively.
Revised: (RainbowMNIST), respectively.

Original: We report the average accuracy with 95% conﬁdence interval over all meta-testing tasks.
Revised: We report the average accuracy with 95% conﬁdence interval over all meta-testing tasks.

Original: For synthetic data, we follow our theoretical analysis and report the crossentropy loss with 95% conﬁdence interval over all meta-testing tasks.
Revised: For synthetic data, we follow our theoretical analysis and report the crossentropy loss with 95% conﬁdence interval over all meta-testing tasks.

Original: The inner-loop and out-loop learning rates are 0.01, 0.001, respectively.
Revised: The inner-loop and out-loop learning rates are 0.01, 0.001, respectively.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Comparison between Different Activation Functions.
Revised: Comparison between Different Activation Functions.

Original: We ﬁrst conduct experiments to compare the performance between Huberized-ReLU activation function used in our theoretical analysis and the traditional ReLU function.
Revised: We ﬁrst conduct experiments to compare the performance between Huberized-ReLU activation function used in our theoretical analysis and the traditional ReLU function.

Original: The results are reported in Table 1, indicating that Huberized-ReLU performs similarly to ReLU.
Revised: The results are reported in Table 1, indicating that Huberized-ReLU performs similarly to ReLU.

Original: Thus it is a reasonable replacement to simplify our analysis.
Revised: Thus it is a reasonable replacement to simplify our analysis.

Original: [Table]
Revised: [Table]

Original: Performance w.r.t.
Revised: Performance w.r.t.

Original: the Number of Inner-Loop Steps.
Revised: the Number of Inner-Loop Steps.

Original: We further compare the performance between the train-train and train-validation methods for different inner-loop optimization steps.
Revised: We further compare the performance between the train-train and train-validation methods for different inner-loop optimization steps.

Original: The results are reported in Table 2.
Revised: The results are reported in Table 2.

Original: According to the results, we observe that the train-validation split performs much better than the train-train method with more inner-loop steps.
Revised: According to the results, we observe that the train-validation split performs much better than the train-train method with more inner-loop steps.

Original: This is what we expected since less number of inner-loop steps corresponds to less overﬁtting to the training set during the inner-loop optimization.
Revised: This is what we expected since less number of inner-loop steps corresponds to less overﬁtting to the training set during the inner-loop optimization.

Original: In the case of using less inner-loop steps, using the training set in the outer-loop optimization can still contribute to the optimization.
Revised: In the case of using less inner-loop steps, using the training set in the outer-loop optimization can still contribute to the optimization.

Original: [Table]
Revised: [Table]

Original: Performance w.r.t.
Revised: Performance w.r.t.

Original: the Inner-Loop Learning Rate and the Outer-Loop Learning Rate.
Revised: the Inner-Loop Learning Rate and the Outer-Loop Learning Rate.

Original: Finally, we analyze the inﬂuence of inner-loop and outer-loop learning rates.
Revised: Finally, we analyze the inﬂuence of inner-loop and outer-loop learning rates.

Original: The results in Table 3 show that train-validation split performs better when the inner-loop learning rate is smaller than the outer-loop learning rate
Revised: The results in Table 3 show that train-validation split performs better when the inner-loop learning rate is smaller than the outer-loop learning rate

Original: [Table]
Revised: [Table]

Original: which corroborates our theoretical results.
Revised: which corroborates our theoretical results.

Original: On synthetic data, the train-train method performs better when the outer-loop learning rate larger than the inner-loop learning rate.
Revised: On synthetic data, the train-train method performs better when the outer-loop learning rate larger than the inner-loop learning rate.

Original: This is not surprising since smaller inner-loop learning rate corresponds to less overﬁtting, similar to the phenomenon when using less inner-loop optimization steps (Table 2)
Revised: This is not surprising since smaller inner-loop learning rate corresponds to less overﬁtting, similar to the phenomenon when using less inner-loop optimization steps (Table 2)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  O VERVIEW OF PROOF TECHNIQUES
Revised:  O VERVIEW OF PROOF TECHNIQUES

Original: From here onward, we will use upper script ( t, τ ) on w j,r to denote the t -th outer-loop iteration and the τ -th inner-loop iteration.
Revised: From here onward, we will use upper script ( t, τ ) on w j,r to denote the t -th outer-loop iteration and the τ -th inner-loop iteration.

Original: However, according to Eq.
Revised: However, according to Eq.

Original: (3.1) and Eq. (3.2), the inner-loop updates for a CNN ﬁlter w j,r is task-speciﬁc.
Revised: (3.1) and Eq. (3.2), the inner-loop updates for a CNN ﬁlter w j,r is task-speciﬁc.

Original: Therefore we cannot rigorously use the notation w ( t,τ ) j,r without referring to a speciﬁc task k .
Revised: Therefore we cannot rigorously use the notation w ( t,τ ) j,r without referring to a speciﬁc task k .

Original: Luckily, all our analyses are based on the study of inner products of the form (cid:104) w j,r , x ( p ) k,i (cid:105) .
Revised: Luckily, all our analyses are based on the study of inner products of the form (cid:104) w j,r , x ( p ) k,i (cid:105) .

Original: We can use the notation (cid:104) w ( t,τ ) j,r , x ( p ) k,i (cid:105) as the inner product after t outer-loop updates followed by τ inner-loop updates, where the inner-loop updates only use samples from task k .
Revised: We can use the notation (cid:104) w ( t,τ ) j,r , x ( p ) k,i (cid:105) as the inner product after t outer-loop updates followed by τ inner-loop updates, where the inner-loop updates only use samples from task k .

Original: With these notations, we give the following two key lemmas that highlight the main differences between the train-train and train-validation methods.
Revised: With these notations, we give the following two key lemmas that highlight the main differences between the train-train and train-validation methods.

Original: Lemma 6.1.
Revised: Lemma 6.1.

Original: Suppose one uses the train-train method to train the neural network.
Revised: Suppose one uses the train-train method to train the neural network.

Original: Let T (1) k,i be the ﬁrst iteration such that max r ∈ [ m ] (cid:104) w ( t, 0)
Revised: Let T (1) k,i be the ﬁrst iteration such that max r ∈ [ m ] (cid:104) w ( t, 0)

Original: j,r , ξ k,i (cid:105) ≥ Ω (cid:0) 1 / polylog( d ) (cid:1) for k ∈ [ K ] , i ∈ [ n ] , and j = y k,i .
Revised: j,r , ξ k,i (cid:105) ≥ Ω (cid:0) 1 / polylog( d ) (cid:1) for k ∈ [ K ] , i ∈ [ n ] , and j = y k,i .

Original: Then, under the same condition as in Theorem 4.4, for any t ≤ T (1) k,i , 1. for any k (cid:48) ∈
Revised: Then, under the same condition as in Theorem 4.4, for any t ≤ T (1) k,i , 1. for any k (cid:48) ∈

Original: [ K ] , r ∈ [ m ] , and j (cid:48) ∈ {− 1 , 1 } we have
Revised: [ K ] , r ∈ [ m ] , and j (cid:48) ∈ {− 1 , 1 } we have

Original: [Equation]
Revised: [Equation]

Original: for r = arg max r
Revised: for r = arg max r

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Lemma 6.2.
Revised: Lemma 6.2.

Original: Suppose one uses the train-validation method to train the neural network.
Revised: Suppose one uses the train-validation method to train the neural network.

Original: Let T (2) j be the ﬁrst iteration such that max r ∈ [ m ] j (cid:104) w ( t, 0) j,r , ν (cid:105) ≥ Ω(1 / polylog( d )) for j ∈ {− 1 , 1 } .
Revised: Let T (2) j be the ﬁrst iteration such that max r ∈ [ m ] j (cid:104) w ( t, 0) j,r , ν (cid:105) ≥ Ω(1 / polylog( d )) for j ∈ {− 1 , 1 } .

Original: Then, under the same condition as in Theorem 4.5, for any t ≤ T (2) j ,
Revised: Then, under the same condition as in Theorem 4.5, for any t ≤ T (2) j ,

Original: for any k ∈ [ K ] and r = arg max r (cid:48) ∈ [ m ] j (cid:104) w ( t, 0) j,r (cid:48) , ν (cid:105) j (cid:104) w ( t,J ) j,r , ν k (cid:105) =
Revised: for any k ∈ [ K ] and r = arg max r (cid:48) ∈ [ m ] j (cid:104) w ( t, 0) j,r (cid:48) , ν (cid:105) j (cid:104) w ( t,J ) j,r , ν k (cid:105) =

Original: Ω(1) j (cid:104) w ( t, 0) j,r , ν k (cid:105) Θ(1) γ J . 2.
Revised: Ω(1) j (cid:104) w ( t, 0) j,r , ν k (cid:105) Θ(1) γ J . 2.

Original: for any r ∈ [ m ] , k ∈ [ K ] , i ∈ I val k , and j (cid:48) ∈ {− 1 , 1 } (cid:104) w ( t,J ) j (cid:48) ,r , ξ k,i (cid:105) ≤ O (cid:16) max (cid:110) (cid:104) w ( t, 0)
Revised: for any r ∈ [ m ] , k ∈ [ K ] , i ∈ I val k , and j (cid:48) ∈ {− 1 , 1 } (cid:104) w ( t,J ) j (cid:48) ,r , ξ k,i (cid:105) ≤ O (cid:16) max (cid:110) (cid:104) w ( t, 0)

Original: j (cid:48) ,r , ξ k,i (cid:105) , J (cid:101)
Revised: j (cid:48) ,r , ξ k,i (cid:105) , J (cid:101)

Original: O ( d − 1 / 2 ) (cid:111)(cid:17) .
Revised: O ( d − 1 / 2 ) (cid:111)(cid:17) .

Original: The above two lemmas give some intuitions for the better generalization power of the train-validation method.
Revised: The above two lemmas give some intuitions for the better generalization power of the train-validation method.

Original: According to Lemma 6.1, the J inner-loop gradient steps amplify the noise inner products more than the feature inner product for the train-train method.
Revised: According to Lemma 6.1, the J inner-loop gradient steps amplify the noise inner products more than the feature inner product for the train-train method.

Original: On the contrary, by Lemma 6.2, the
Revised: On the contrary, by Lemma 6.2, the

Original: J inner-loop gradient steps amplify the feature inner product and do not have a big impact on the noise inner products for the train-validation method.
Revised: J inner-loop gradient steps amplify the feature inner product and do not have a big impact on the noise inner products for the train-validation method.

Original: In the rest of this section, we mainly sketch the proof of Theorem 4.5 for the train-validation method.
Revised: In the rest of this section, we mainly sketch the proof of Theorem 4.5 for the train-validation method.

Original: Based on Lemma 6.2, we can see that the inner-loop updates in the train-validation method prioritizes feature learning over noise memorization.
Revised: Based on Lemma 6.2, we can see that the inner-loop updates in the train-validation method prioritizes feature learning over noise memorization.

Original: Our further study of the outer-loop training procedure gives the following lemma.
Revised: Our further study of the outer-loop training procedure gives the following lemma.

Original: Lemma 6.3.
Revised: Lemma 6.3.

Original: Under the same condition as in Theorem 4.5, let T (2) =
Revised: Under the same condition as in Theorem 4.5, let T (2) =

Original: max j T (2) j , where T (2) j is deﬁned in Lemma 6.2.
Revised: max j T (2) j , where T (2) j is deﬁned in Lemma 6.2.

Original: Then max r ∈ [ m ] (cid:104) w ( T (2) , 0) j,r , ξ k,i (cid:105)
Revised: Then max r ∈ [ m ] (cid:104) w ( T (2) , 0) j,r , ξ k,i (cid:105)

Original: =
Revised: =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( d − 1 / 2 ) for any k ∈ [ K ] , i ∈ I
Revised: ( d − 1 / 2 ) for any k ∈ [ K ] , i ∈ I

Original: val k and j ∈ {− 1 , 1 } .
Revised: val k and j ∈ {− 1 , 1 } .

Original: We remind our readers that T (2) (deﬁned in Lemma 6.2) represents the time taken for the feature inner product to grow to Ω(1 / polylog( d )) .
Revised: We remind our readers that T (2) (deﬁned in Lemma 6.2) represents the time taken for the feature inner product to grow to Ω(1 / polylog( d )) .

Original: Therefore Lemma 6.3 is essentially still a comparison between the growth rate of feature and noise inner products: it shows that when the feature inner product grows to (cid:101) Θ(1) , the noise inner products still remain at their initialization order which is (cid:101)
Revised: Therefore Lemma 6.3 is essentially still a comparison between the growth rate of feature and noise inner products: it shows that when the feature inner product grows to (cid:101) Θ(1) , the noise inner products still remain at their initialization order which is (cid:101)

Original: O
Revised: O

Original: ( d − 1 / 2 ) .
Revised: ( d − 1 / 2 ) .

Original: The next lemma provides a key result for the convergence of the training loss.
Revised: The next lemma provides a key result for the convergence of the training loss.

Original: Lemma 6.4.
Revised: Lemma 6.4.

Original: Let T = poly( d ) be the total number of iterations.
Revised: Let T = poly( d ) be the total number of iterations.

Original: Under the same condition as in
Revised: Under the same condition as in

Original: Theorem 4.5, we have
Revised: Theorem 4.5, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) (cid:48) ( t,J ) k,i = (cid:96)
Revised: where (cid:96) (cid:48) ( t,J ) k,i = (cid:96)

Original: [Equation]
Revised: [Equation]

Original: W ( t ) , S tr k ) , x k,i ))
Revised: W ( t ) , S tr k ) , x k,i ))

Original: By deﬁnition, the feature inner product has grown to Ω(1 / polylog( d )) at time T (2) .
Revised: By deﬁnition, the feature inner product has grown to Ω(1 / polylog( d )) at time T (2) .

Original: After T (2) , we implement a more careful study of the training process, as some data points may have been wellﬁtted (with a small loss) and no longer signiﬁcantly contribute to the training.
Revised: After T (2) , we implement a more careful study of the training process, as some data points may have been wellﬁtted (with a small loss) and no longer signiﬁcantly contribute to the training.

Original: Our next theorem shows that using the train-validation method, after T (2) , the feature inner product will grow even larger and will be at least Ω( K c ) for some c > 0 at the end of the training.
Revised: Our next theorem shows that using the train-validation method, after T (2) , the feature inner product will grow even larger and will be at least Ω( K c ) for some c > 0 at the end of the training.

Original: Lemma 6.5.
Revised: Lemma 6.5.

Original: Under the same condition as in Theorem 4.5, suppose one runs the train-validation method for a total of T = poly( d ) iterations.
Revised: Under the same condition as in Theorem 4.5, suppose one runs the train-validation method for a total of T = poly( d ) iterations.

Original: Then max r ∈ [ m ] j (cid:104) w ( T, 0) j,r , ν (cid:105) ≥ Ω( K c ) for all j ∈ {− 1 , 1 } and for some c > 0 .
Revised: Then max r ∈ [ m ] j (cid:104) w ( T, 0) j,r , ν (cid:105) ≥ Ω( K c ) for all j ∈ {− 1 , 1 } and for some c > 0 .

Original: Lemma 6.5 shows that at the end of training, the neural network has sufﬁciently learned the shared feature ν .
Revised: Lemma 6.5 shows that at the end of training, the neural network has sufﬁciently learned the shared feature ν .

Original: When given a fresh task, we should expect a small test loss because the shared feature ν will also be present in this newly sampled task.
Revised: When given a fresh task, we should expect a small test loss because the shared feature ν will also be present in this newly sampled task.

Original: We are now ready to present the proof of Theorem 4.5.
Revised: We are now ready to present the proof of Theorem 4.5.

Original: Proof of Theorem 4.5.
Revised: Proof of Theorem 4.5.

Original: By deﬁnition, (cid:98) L tr − val ( W ( t ) , {S k } Kk =1 ) =
Revised: By deﬁnition, (cid:98) L tr − val ( W ( t ) , {S k } Kk =1 ) =

Original: (cid:80)
Revised: (cid:80)

Original: k ∈ [ K ] (cid:80) i ∈I val k
Revised: k ∈ [ K ] (cid:80) i ∈I val k

Original: (cid:96) ( t,J ) k,i , where (cid:96) ( t,J ) k,i = (cid:96) ( y k,i · f ( (cid:101) W ( W ( t ) , S tr k ) , x k,i )) .
Revised: (cid:96) ( t,J ) k,i , where (cid:96) ( t,J ) k,i = (cid:96) ( y k,i · f ( (cid:101) W ( W ( t ) , S tr k ) , x k,i )) .

Original: Then by Lemma 6.4 and the property of the cross-entropy loss that − (cid:96) (cid:48) ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 , we have min t ∈
Revised: Then by Lemma 6.4 and the property of the cross-entropy loss that − (cid:96) (cid:48) ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 , we have min t ∈

Original: [ T ] (cid:98) L tr − val
Revised: [ T ] (cid:98) L tr − val

Original: [Equation]
Revised: [Equation]

Original: This proves the ﬁrst part of Theorem 4.5.
Revised: This proves the ﬁrst part of Theorem 4.5.

Original: For the second part, consider a new data point ( x , y ) .
Revised: For the second part, consider a new data point ( x , y ) .

Original: By
Revised: By

Original: Deﬁnition 3.1 and Deﬁnition 3.2, the new data input x consists of two patches, one of which is a noise vector ξ that is generated from N ( 0 , σ 2 ξ · ( I − P )) .
Revised: Deﬁnition 3.1 and Deﬁnition 3.2, the new data input x consists of two patches, one of which is a noise vector ξ that is generated from N ( 0 , σ 2 ξ · ( I − P )) .

Original: Denote by E the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w
Revised: Denote by E the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w

Original: [Equation]
Revised: [Equation]

Original: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈
Revised: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈

Original: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } , where ξ is the noise vector from the new data x .
Revised: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } , where ξ is the noise vector from the new data x .

Original: Then using Gaussian concentration, we have P ( E ) ≥ 1 − (cid:101) O
Revised: Then using Gaussian concentration, we have P ( E ) ≥ 1 − (cid:101) O

Original: (exp( − d 1 / 4 )) .
Revised: (exp( − d 1 / 4 )) .

Original: We divide L test ( W ( T ) ) into two parts:
Revised: We divide L test ( W ( T ) ) into two parts:

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122) (cid:125) I.
Revised: (cid:123)(cid:122) (cid:125) I.

Original: From this decomposition, it is clear that bounding I 1 is more important, since P ( E c ) is exponentially small, which makes I 2 small.
Revised: From this decomposition, it is clear that bounding I 1 is more important, since P ( E c ) is exponentially small, which makes I 2 small.

Original: Indeed, one can bound I 

Revised: Indeed, one can bound I

Original: = O (poly( d )) exp( − 0 .
Revised: = O (poly( d )) exp( − 0 .

Original: 5 d 1 / 4 ) .
Revised: 5 d 1 / 4 ) .

Original: Moreover, under the event E , it holds that F − y ( W ( T ) − y , x ) < log(2) , and F y ( W ( T ) y , x ) ≥ σ (cid:0) max r ∈ [ m ] y · (cid:104) w ( T, 0) y,r , ν (cid:105) (cid:1) .
Revised: Moreover, under the event E , it holds that F − y ( W ( T ) − y , x ) < log(2) , and F y ( W ( T ) y , x ) ≥ σ (cid:0) max r ∈ [ m ] y · (cid:104) w ( T, 0) y,r , ν (cid:105) (cid:1) .

Original: Applying Lemma 6.5, we have
Revised: Applying Lemma 6.5, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by the deﬁnition of (cid:96) ( · ) and the inequality log(1+ x ) ≤ x, ∀ x ≥ 0 .
Revised: where the last inequality is by the deﬁnition of (cid:96) ( · ) and the inequality log(1+ x ) ≤ x, ∀ x ≥ 0 .

Original: Therefore we have that I 1 ≤ 2 exp( − K c ) .
Revised: Therefore we have that I 1 ≤ 2 exp( − K c ) .

Original: Combining the bounds on I 1 and I 2 yields L test ( W ( T ) ) ≤ 2 exp( − K c )
Revised: Combining the bounds on I 1 and I 2 yields L test ( W ( T ) ) ≤ 2 exp( − K c )

Original: + O (poly( d )) exp( − 0 .
Revised: + O (poly( d )) exp( − 0 .

Original: 5 d 1 / 4 )
Revised: 5 d 1 / 4 )

Original: =
Revised: =

Original: O (cid:0) exp( − K c ) (cid:1) , which proves the theorem.
Revised: O (cid:0) exp( − K c ) (cid:1) , which proves the theorem.

Original:  C ONCLUSION AND F UTURE WORK
Revised:  C ONCLUSION AND F UTURE WORK

Original: In this work, we study the FOMAML algorithm applied to a classiﬁcation problem with a two-layer
Revised: In this work, we study the FOMAML algorithm applied to a classiﬁcation problem with a two-layer

Original: CNN trained by gradient descent.
Revised: CNN trained by gradient descent.

Original: We proved that although both train-train and train-validation methods can achieve a small training loss, to get good generalization results, it is necessary to perform a train-validation split in the data when the noise is large and the number of samples is limited.
Revised: We proved that although both train-train and train-validation methods can achieve a small training loss, to get good generalization results, it is necessary to perform a train-validation split in the data when the noise is large and the number of samples is limited.

Original: It is of interest to extend our result to more other types of data model (for example, natural language data), and deeper neural network structures.
Revised: It is of interest to extend our result to more other types of data model (for example, natural language data), and deeper neural network structures.

Original: We would also like to extend our analysis to other popular meta-learning algorithms such as MAML (with hessian term), iMAML (Rajeswaran et al., 2019),
Revised: We would also like to extend our analysis to other popular meta-learning algorithms such as MAML (with hessian term), iMAML (Rajeswaran et al., 2019),

Original: Meta-MiniBatchProx (Zhou et al., 2019), Reptile (Nichol et al., 2018) and closed-form solvers (Bertinetto et al., 2018).
Revised: Meta-MiniBatchProx (Zhou et al., 2019), Reptile (Nichol et al., 2018) and closed-form solvers (Bertinetto et al., 2018).

Original: A C OMPARISON WITH B AI ET AL . (2021)
Revised: A C OMPARISON WITH B AI ET AL . (2021)

Original: In this section, we point out some differences between our experiment setup and that of Bai et al.
Revised: In this section, we point out some differences between our experiment setup and that of Bai et al.

Original: (2021), which also studies the importance of a train-validation split in meta-learning.
Revised: (2021), which also studies the importance of a train-validation split in meta-learning.

Original: The main conclusion of Bai et al.
Revised: The main conclusion of Bai et al.

Original: (2021) is that the train-train method could outperform the train-validation method asymptotically under a linear centroid model.
Revised: (2021) is that the train-train method could outperform the train-validation method asymptotically under a linear centroid model.

Original: They have also run experiments using CNN as backbone to support their theory (see Table 1 of Bai et al.
Revised: They have also run experiments using CNN as backbone to support their theory (see Table 1 of Bai et al.

Original: This may seem to contradict our experiments at ﬁrst sight, since we showed in Table 1 that the train-validation method should outperform the train-train method using CNN as backbone.
Revised: This may seem to contradict our experiments at ﬁrst sight, since we showed in Table 1 that the train-validation method should outperform the train-train method using CNN as backbone.

Original: We would like to point out that this is due to the different choice of loss function and optimization algorithms used by us and Bai et al.
Revised: We would like to point out that this is due to the different choice of loss function and optimization algorithms used by us and Bai et al.

Original: (2021) as explained as follows.
Revised: (2021) as explained as follows.

Original: A. R EGULARIZER IN INNER - LOOP
Revised: A. R EGULARIZER IN INNER - LOOP

Original: Let us focus on the train-train method.
Revised: Let us focus on the train-train method.

Original: Our loss function is given by
Revised: Our loss function is given by

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) W ( W , S k ) represents the weights of the network after J gradient descent steps (w.r.t.
Revised: where (cid:101) W ( W , S k ) represents the weights of the network after J gradient descent steps (w.r.t.

Original: loss
Revised: loss

Original: L ( · , S k ) ) starting from W .
Revised: L ( · , S k ) ) starting from W .

Original: While Bai et al.
Revised: While Bai et al.

Original: (2021) has their loss function given by
Revised: (2021) has their loss function given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where λ is a regularization parameter.
Revised: where λ is a regularization parameter.

Original: This regularizer is popularized by Rajeswaran et al.
Revised: This regularizer is popularized by Rajeswaran et al.

Original: and Zhou et al.
Revised: and Zhou et al.

Original: In practice, however, many meta-learning methods do not use this regularizer in the inner-loop (add citations).
Revised: In practice, however, many meta-learning methods do not use this regularizer in the inner-loop (add citations).

Original: A. E XPERIMENT ALGORITHMS
Revised: A. E XPERIMENT ALGORITHMS

Original: To compare the performance of train-validation and train-train methods, Bai et al.
Revised: To compare the performance of train-validation and train-train methods, Bai et al.

Original: (2021) used iMAML (Rajeswaran et al., 2019) for the train-validation method and
Revised: (2021) used iMAML (Rajeswaran et al., 2019) for the train-validation method and

Original: Meta-MiniBatchProx
Revised: Meta-MiniBatchProx

Original: (Zhou et al., 2019) for the train-train method.
Revised: (Zhou et al., 2019) for the train-train method.

Original: Both algorithms were developed to target the loss function given by Eq.
Revised: Both algorithms were developed to target the loss function given by Eq.

Original: (A.1).
Revised: (A.1).

Original: In particular, iMAML used implicit function theorem to calculate the gradient ∇ W (cid:98) W ( W , S k ) , and then uses this information to do gradient descent on W with respect to the loss function given by Eq.
Revised: In particular, iMAML used implicit function theorem to calculate the gradient ∇ W (cid:98) W ( W , S k ) , and then uses this information to do gradient descent on W with respect to the loss function given by Eq.

Original: (A.1).
Revised: (A.1).

Original: iMAML is similar to MAML in the sense that both algorithms perform gradient descent on W with respect to their respective loss functions (cid:98) L tr − tr .
Revised: iMAML is similar to MAML in the sense that both algorithms perform gradient descent on W with respect to their respective loss functions (cid:98) L tr − tr .

Original: However, the structure of Meta-MiniBatchProx is different from that of iMAML and MAML.
Revised: However, the structure of Meta-MiniBatchProx is different from that of iMAML and MAML.

Original: It is more closely related to
Revised: It is more closely related to

Original: Reptile (Nichol et al., 2018), which is another ﬁrst order meta-learning algorithm.
Revised: Reptile (Nichol et al., 2018), which is another ﬁrst order meta-learning algorithm.

Original: Instead of performing gradient descent in the outer-loop, Reptile updates W at each step as a convex combination of W and (cid:101) W ( W , S k ) .
Revised: Instead of performing gradient descent in the outer-loop, Reptile updates W at each step as a convex combination of W and (cid:101) W ( W , S k ) .

Original: This gives
Revised: This gives

Original: [Equation]
Revised: [Equation]

Original: for some ε ∈ (0 , 1) .
Revised: for some ε ∈ (0 , 1) .

Original: And Meta-MiniBatchProx replaces (cid:101) W ( W , S k ) with (cid:98) W ( W , S k ) in the above.
Revised: And Meta-MiniBatchProx replaces (cid:101) W ( W , S k ) with (cid:98) W ( W , S k ) in the above.

Original: In our experiments, we used FOMAML for both train-validation and train-train.
Revised: In our experiments, we used FOMAML for both train-validation and train-train.

Original: We believe that by using the same algorithm for train-validation with train-train, the comparison is fairer.
Revised: We believe that by using the same algorithm for train-validation with train-train, the comparison is fairer.

Original: B A DDITIONAL E XPERIMENTS B. C OMPARISON WITH R EPTILE (N ICHOL ET AL ., 2018)
Revised: B A DDITIONAL E XPERIMENTS B. C OMPARISON WITH R EPTILE (N ICHOL ET AL ., 2018)

Original: We compare FOMAML (using the train-validation method) with Reptile, which is another ﬁrstorder algorithm used in meta-learning.
Revised: We compare FOMAML (using the train-validation method) with Reptile, which is another ﬁrstorder algorithm used in meta-learning.

Original: Different from FOMAML, Reptile does not require a trainvalidation split in the data set.
Revised: Different from FOMAML, Reptile does not require a trainvalidation split in the data set.

Original: In addition, the outer-loop of Reptile does not perform gradient descent, but use a convex combination between the unadapted weights and the task-adapted weights (see Eq.
Revised: In addition, the outer-loop of Reptile does not perform gradient descent, but use a convex combination between the unadapted weights and the task-adapted weights (see Eq.

Original: (A.3)).
Revised: (A.3)).

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: We see that on both RainbowMNIST and miniImagenet, FOMAML with train-validation outperforms Reptile by a small margin.
Revised: We see that on both RainbowMNIST and miniImagenet, FOMAML with train-validation outperforms Reptile by a small margin.

Original: [Table]
Revised: [Table]

Original: B. T IME EVOLUTION OF TEST LOSS OF FOMAML
Revised: B. T IME EVOLUTION OF TEST LOSS OF FOMAML

Original: We plot the time evolution of test loss using trained by FOMAML with train-validation and with train-train on our synthetic data set.
Revised: We plot the time evolution of test loss using trained by FOMAML with train-validation and with train-train on our synthetic data set.

Original: The results are illustrated in Figure 1a and 1b for train-validation and train-train, respectively.
Revised: The results are illustrated in Figure 1a and 1b for train-validation and train-train, respectively.

Original: [Figure]
Revised: [Figure]

Original: B. FOMAML WITH DIFFERENT NEURAL NETWORKS
Revised: B. FOMAML WITH DIFFERENT NEURAL NETWORKS

Original: In this section [Figure] we show that FOMAML with train-validaiton still outperforms FOMAML with traintrain on CNN with more layers and ResNet.
Revised: In this section, we show that FOMAML with train-validaiton still outperforms FOMAML with train-train on CNN with more layers and ResNet.

Original: The experiments are run on miniImagenet.
Revised: The experiments are run on miniImagenet.

Original: The results are recorded in Table 5.
Revised: The results are recorded in Table 5.

Original: We see that FOMAML with train-validation outperforms FOMAML with train-train by a large margin under all 3 neural network structures.
Revised: We see that FOMAML with train-validation outperforms FOMAML with train-train by a large margin under all 3 neural network structures.

Original: [Table]
Revised: [Table]

Original: B. E FFECT OF THE NUMBER OF SAMPLES PER TASK
Revised: E FFECT OF THE NUMBER OF SAMPLES PER TASK

Original: We then discuss the performance with respect to the number of examples.
Revised: We then discuss the performance with respect to the number of examples.

Original: In Table 6, we report the performance on miniImagenent when the number of examples per class is 1, 3, 5, which corresponds to 5-way 1-shot, 5-way 3-shot and 5-way 5-shot settings.
Revised: In Table 6, we report the performance on miniImagenent when the number of examples per class is 1, 3, 5, which corresponds to 5-way 1-shot, 5-way 3-shot and 5-way 5-shot settings.

Original: We observe that as the number of samples per class increases, both the train-train and train-validation methods receive a performance increase. 
Revised: We observe that as the number of samples per class increases, both the train-train and train-validation methods receive a performance increase. [Table]

Original: C P RELIMINARY CALCULATIONS
Revised: C P RELIMINARY CALCULATIONS

Original: In this section, we present some calculations that are useful for our derivations later.
Revised: In this section [Equation] we present some calculations that are useful for our derivations later.

Original: Recall that (cid:96) ( z ) =
Revised: Recall that (cid:96) ( z ) =

Original: log(1+exp( − z )) , which gives (cid:96) (cid:48) ( z ) = − (1+exp( z )) − 1 .
Revised: log(1+exp( − z )) , which gives (cid:96) (cid:48) ( z ) = − (1+exp( z )) − 1 .

Original: Then, we can compute the partial[Table] derivative of the loss function using samples from task k as
Revised: Then, we can compute the partial derivative of the loss function using samples from task k as

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: if we only evaluate the loss on the training set of S k we have
Revised: if we only evaluate the loss on the training set of S k we have

Original: [Equation]
Revised: [Equation]

Original: Let t ≥ 0 and τ ∈ [0 , J ] be the indices for the outer-loop and inner-loop respectively.
Revised: Let t ≥ 0 and τ ∈ [0 , J ] be the indices for the outer-loop and inner-loop respectively.

Original: We remind the readers our notation.
Revised: We remind the readers our notation.

Original: Denote (cid:96) (cid:48) ( t,J ) k,i = (cid:96) (cid:48) ( y k,i · f ( (cid:101) W ( W ( t ) , S k ) , x k,i )) when referring to the traintrain method and (cid:96) (cid:48) ( t,J ) k,i = (cid:96) (cid:48) ( y k,i · f ( (cid:101) W ( W ( t ) , S tr k ) , x k,i )) when referring to the train-validation method.
Revised: Denote (cid:96) (cid:48) ( t,J ) k,i = (cid:96) (cid:48) ( y k,i · f ( (cid:101) W ( W ( t ) , S k ) , x k,i )) when referring to the traintrain method and (cid:96) (cid:48) ( t,J ) k,i = (cid:96) (cid:48) ( y k,i · f ( (cid:101) W ( W ( t ) , S tr k ) , x k,i )) when referring to the train-validation method.

Original: At outer step t , using the samples from task k , we have the inner-loop updates as
Revised: At outer step t , using the samples from task k , we have the inner-loop updates as

Original: [Equation]
Revised: [Equation]

Original: For noise ξ k,i , we have the inner-loop updates as (suppose we are using all the samples from task k for the inner-loop optimization)
Revised: For noise ξ k,i , we have the inner-loop updates as (suppose we are using all the samples from task k for the inner-loop optimization)

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst sum goes from i (cid:48)
Revised: where the ﬁrst sum goes from i (cid:48)

Original: = 1 to i (cid:48)
Revised: = 1 to i (cid:48)

Original: =
Revised: =

Original: n , omitting the term when i (cid:48) =
Revised: n , omitting the term when i (cid:48) =

Original: i .
Revised: i .

Original: If we consider a noise ξ k,i from the validation set of some task,
Revised: If we consider a noise ξ k,i from the validation set of some task,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e [Equation] k ∈
Revised: e. k ∈

Original: [ K ] and i ∈ I val k , and we only use the training set of task k for inner-loop updates, then
Revised: [ K ] and i ∈ I val k , and we only use the training set of task k for inner-loop updates, then

Original: [Equation]
Revised: [Equation]

Original: If we consider a noise ξ k,i from the training set of a task, i
Revised: If we consider a noise ξ k,i from the training set of a task, i

Original: [Equation]
Revised: [Equation]

Original: e. k ∈ [ K ] and i ∈ I tr k , and we only use the training set of task k for inner-loop updates, then
Revised: e. k ∈ [ K ] and i ∈ I tr k , and we only use the training set of task k for inner-loop updates, then

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: C.

Revised: [Equation]

Original: where the sum goes from i (cid:48)
Revised: where the sum goes from i (cid:48)

Original: = 1 to i (cid:48)
Revised: = 1 to i (cid:48)

Original: = n 1 except when i (cid:48)
Revised: = n 1 except when i (cid:48)

Original: =
Revised: =

Original: i .
Revised: i .

Original: Using only the training set of task k [Equation] the feature inner product update is
Revised: Using only the training set of task k , the feature inner product update is

Original: [Equation]
Revised: [Equation]

Original: Next we look at the outer-loop updates.
Revised: Next we look at the outer-loop updates.

Original: For the train-train method, the noise update is
Revised: For the train-train method, the noise update is

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: where the sum is over all k (
Revised: where the sum is over all k (

Original: cid:48) ∈
Revised: cid:48) ∈

Original: [ K ] and i (cid:48) ∈ [ n ] except when ( k (cid:48) , [Table] ) =
Revised: [ K ] and i (cid:48) ∈ [ n ] except when ( k (cid:48) , i (cid:48) ) =

Original: ( k, i ) .
Revised: ( k, i ) .

Original: Hence, the number of terms in the sum is Kn − 1 .
Revised: Hence, the number of terms in the sum is Kn − 1 .

Original: And
Revised: And

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The arguments in σ (cid:48) ( · ) in Eq.
Revised: The arguments in σ (cid:48) ( · ) in Eq.

Original: (C.6) are given by Eq.
Revised: (C.6) are given by Eq.

Original: (C.2).
Revised: (C.2).

Original: The outer-loop feature update is
Revised: The outer-loop feature update is

Original: [Equation]
Revised: [Equation]

Original: where the arguments in σ (cid:48) ( · ) are given by Eq.
Revised: where the arguments in σ (cid:48) ( · ) are given by Eq.

Original: (C.1).
Revised: (C.1).

Original: The arguments in σ ( · ) in Eq.
Revised: The arguments in σ ( · ) in Eq.

Original: (C.7)) are given by Eq.
Revised: (C.7)) are given by Eq.

Original: (C.2) and Eq.
Revised: (C.2) and Eq.

Original: (C.1).
Revised: (C.1).

Original: For the train-validation method, consider a noise ξ k,i from the validation set of the k -th task, i.e. i ∈
Revised: For the train-validation method, consider a noise ξ k,i from the validation set of the k -th task, i.e. i ∈

Original: I val k .
Revised: I val k .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the sum is over all k (
Revised: where the sum is over all k (

Original: cid:48) ∈
Revised: cid:48) ∈

Original: [ K ] and i (cid:48) ∈ I val k except when ( k (cid:48) , i (cid:48) )
Revised: [ K ] and i (cid:48) ∈ I val k except when ( k (cid:48) , i (cid:48) )

Original: =
Revised: =

Original: ( k, i ) .
Revised: ( k, i ) .

Original: The arguments in σ (cid:48) ( · ) are given by Eq.
Revised: The arguments in σ (cid:48) ( · ) are given by Eq.

Original: (C.3).
Revised: (C.3).

Original: If we consider a noise ξ k,i from the training set, i.e. k ∈ [ K ] and i ∈ I tr k , then the outer-loop update is given by
Revised: If we consider a noise ξ k,i from the training set, i.e. k ∈ [ K ] and i ∈ I tr k , then the outer-loop update is given by

Original: [Equation]
Revised: [Equation]

Original: 10) where the sum is over all k ∈
Revised: where the sum is over all k ∈

Original: [ K ] and i ∈ I val k .
Revised: [ K ] and i ∈ I val k .

Original: And the arguments in σ (cid:48) ( · ) are given by Eq.
Revised: And the arguments in σ (cid:48) ( · ) are given by Eq.

Original: (C.4).
Revised: (C.4).

Original: The outer-loop update of the feature using the train-validation method is similar to using the traintrain method:
Revised: The outer-loop update of the feature using the train-validation method is similar to using the traintrain method:

Original: [Equation]
Revised: [Equation]

Original: (C [Equation] 11) where the arguments in σ (cid:48) ( · ) in Eq.
Revised: where the arguments in σ (cid:48) ( · ) in Eq.

Original: (C.11) are given by Eq.
Revised: (C.11) are given by Eq.

Original: (C.5).
Revised: (C.5).

Original: Note that the (cid:96) (cid:48) term in Eq.
Revised: Note that the (cid:96) (cid:48) term in Eq.

Original: (C.9),
Revised: (C.9),

Original: Eq.
Revised: Eq.

Original: (C.10) and Eq.
Revised: (C.10) and Eq.

Original: (C.11) are given by Eq.
Revised: (C.11) are given by Eq.

Original: (C.7), but the arguments in σ ( · ) in Eq.
Revised: (C.7), but the arguments in σ ( · ) in Eq.

Original: (C.7)) are now given by Eq.
Revised: (C.7)) are now given by Eq.

Original: (C.3) and Eq.
Revised: (C.3) and Eq.

Original: (C.5).
Revised: (C.5).

Original: D P RELIMINARY L EMMAS
Revised: D P RELIMINARY L EMMAS

Original: We will work under the following parameters for the rest of the proof.
Revised: We will work under the following parameters for the rest of the proof.

Original: Condition D.1.
Revised: Condition D.1.

Original: (cid:107) ν(cid:107) 2 = 1 , σ 0 = 1 √ d , σ s = [Equation] [Equation] σ
Revised: (cid:107) ν

Original: [Equation]
Revised: [Equation]

Original: , n = Θ(1) , K = log( d )

Original: [Equation]
Revised: [Equation]

Original:  , m = log( d ) 0 .
Revised: 5 , m = log( d ) 0 .

Original: 2 , η = 1 log( d ) − 5 , γ = log( d ) 0 .
Revised: 2 , η = 1 log( d ) − 5 , γ = log( d ) 0 .

Original: 4 , J = 5 , h = 12 , M = d/ 2 .
Revised: 4 , J = 5 , h = 12 , M = d/ 2 .

Original: We will also assume that our data split is symmetric: for each k ∈ [ K ] , and j ∈ {− 1 , 1 } (cid:12) (cid:12) { i : i ∈ I tr k , y k,i =
Revised: We will also assume that our data split is symmetric: for each k ∈ [ K ] , and j ∈ {− 1 , 1 } (cid:12)(cid:12) { i : i ∈ I tr k , y k,i =

Original: j } (cid:12) (cid:12) > 0 , (cid:12) [Equation] [Equation] val
Revised: j } (cid:12)(cid:12) > 0 , (cid:12)(cid:12) { i : i ∈

Original: [Equation] ,i [Equation] } (cid:12) (cid:12) > 0 .
Revised: I val k , y k,i = j } (cid:12)(cid:12) > 0 .

Original: Lemma D.2.
Revised: Lemma D.2.

Original: Under Condition D.1 and Condition 4.1, the following estimates hold with probability at least 1 − ( Kn ) − 
Revised: Under Condition D.1 and Condition 4.1, the following estimates hold with probability at least 1 − ( Kn ) − 10 .

Original: (cid:107) ξ k,i (cid:107) 22 =
Revised: (cid:107) ξ k,i (cid:107) 22 =

Original: Θ( σ 2 ξ d ) for any k ∈ [ K ] and i ∈ [ n ] .
Revised: Θ( σ 2 ξ d ) for any k ∈ [ K ] and i ∈ [ n ] .

Original: max (cid:8) |(cid:104) ξ k,i , ξ k (cid:48) ,i (cid:48)
Revised: max (cid:8) |(cid:104) ξ k,i , ξ k (cid:48) ,i (cid:48)

Original: (cid:105)| : k, k (cid:48) ∈
Revised: (cid:105)| : k, k (cid:48) ∈

Original: [ K ] , i, i (cid:48) ∈ [ n ] , ( k, i ) (cid:54) =
Revised: [ K ] , i, i (cid:48) ∈ [ n ] , ( k, i ) (cid:54) =

Original: ( k (cid:48) , i (cid:48) )
Revised: ( k (cid:48) , i (cid:48) )

Original: (cid:9) ≤
Revised: (cid:9) ≤

Original: O ( σ 2 ξ √ d (cid:112) log( Kn )) .
Revised: O ( σ 2 ξ √ d (cid:112) log( Kn )) .

Original: 3. max (cid:8) |(cid:104) w (0 , 0) j,r , ν (cid:105)| : r ∈ [ m ] (cid:9) =
Revised: max (cid:8) |(cid:104) w (0 , 0) j,r , ν (cid:105)| : r ∈ [ m ] (cid:9) =

Original: Θ( σ 0 (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } .
Revised: Θ( σ 0 (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } .

Original: max (cid:8) |(cid:104) w (0 , 0) j,r , ξ k,i (cid:105)| : r ∈ [ m ] (cid:9) =
Revised: max (cid:8) |(cid:104) w (0 , 0) j,r , ξ k,i (cid:105)| : r ∈ [ m ] (cid:9) =

Original: Θ( σ 0 σ ξ √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈
Revised: Θ( σ 0 σ ξ √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈

Original: [ K ] , i ∈ [ n ] .
Revised: [ K ] , i ∈ [ n ] .

Original: Ω( σ 0 ) ≤ max r ∈ [ m ] j (cid:104) w (0 , 0) j,r , ν (cid:105) ≤
Revised: Ω( σ 0 ) ≤ max r ∈ [ m ] j (cid:104) w (0 , 0) j,r , ν (cid:105) ≤

Original: O ( σ 0 (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } .
Revised: O ( σ 0 (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } .

Original: Ω( σ 0 σ ξ √ d ) ≤ max r ∈ [ m ] (cid:104) w (0 , 0) j,r , ξ k,i (cid:105) ≤
Revised: Ω( σ 0 σ ξ √ d ) ≤ max r ∈ [ m ] (cid:104) w (0 , 0) j,r , ξ k,i (cid:105) ≤

Original: O ( σ 0 σ ξ √ d (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } , k ∈
Revised: O ( σ 0 σ ξ √ d (cid:112) log( mKn )) for any j ∈ {− 1 , 1 } , k ∈

Original: [ K ] , i ∈ [ n ] .
Revised: [ K ] , i ∈ [ n ] .

Original: max (cid:8) |(cid:104) w (0 , 0) j,r , z k (cid:105)| : r ∈ [ m ] (cid:9) =
Revised: max (cid:8) |(cid:104) w (0 , 0) j,r , z k (cid:105)| : r ∈ [ m ] (cid:9) =

Original: Θ( σ 0 σ s √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈
Revised: Θ( σ 0 σ s √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈

Original: [ K ] . 8.
Revised: [ K ] . 8.

Original: Ω( σ 0 σ s √ d ) ≤ max r ∈ [ m ] j (cid:104) w (0 , 0) j,r , z k (cid:105) ≤
Revised: Ω( σ 0 σ s √ d ) ≤ max r ∈ [ m ] j (cid:104) w (0 , 0) j,r , z k (cid:105) ≤

Original: O ( σ 0 σ s √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈
Revised: O ( σ 0 σ s √ d (cid:112) log ( mKn )) for any j ∈ {− 1 , 1 } , k ∈

Original: [ M ] .
Revised: [ M ] .

Original: Proof.
Revised: Proof.

Original: By Lemma G. [Table] we have with probability at least 1 − δ/ 3 ,
Revised: By Lemma G.1, we have with probability at least 1 − δ/ 3 ,

Original: [Equation]
Revised: [Equation]

Original: for all k, k (cid:48) ∈
Revised: for all k, k (cid:48) ∈

Original: [ K ] and i, i (cid:48) ∈ [ n ] with ( k, i ) (cid:54) =
Revised: [ K ] and i, i (cid:48) ∈ [ n ] with ( k, i ) (cid:54) =

Original: ( k (cid:48) , i (cid:48) ) .
Revised: ( k (cid:48) , i (cid:48) ) .

Original: By Lemma G.2, we have with probability at least 1 − δ/ 3 ,
Revised: By Lemma G.2, we have with probability at least 1 − δ/ 3 ,

Original: [Equation]
Revised: [Equation]

Original: for all r ∈ [ m ] , j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .
Revised: for all r ∈ [ m ] , j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: for all j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .
Revised: for all j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .

Original: Using similar ideas, we get that with probability at least 1 − δ/ 3 ,|
Revised: Using similar ideas, we get that with probability at least 1 − δ/ 3 ,

Original: [Equation]
Revised: [Equation]

Original: w
Revised: d

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ,r , z k

Original: [Equation]
Revised: [Equation]

Original: for any k ∈ [ K ] , r ∈ [ m ] , and j ∈ {− 1 , 1 } .
Revised: for any k ∈ [ K ] , r ∈ [ m ] , and j ∈ {− 1 , 1 } .

Original: The second inequality is by Condition 4.1.
Revised: The second inequality is by Condition 4.1.

Original: The last inequality is again by Lemma G.2.
Revised: The last inequality is again by Lemma G.2.

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: Combining the above results, we get that all of the above events hold simultaneously with probability at least 1 − δ .
Revised: Combining the above results, we get that all of the above events hold simultaneously with probability at least 1 − δ .

Original: Taking δ =
Revised: Taking δ =

Original: ( Kn ) − 10 gives the desired result.
Revised: ( Kn ) − 10 gives the desired result.

Original: Remark D.3 .
Revised: Remark D.3 .

Original: Since we will use the estimates in Lemma D.2 in the rest of our proofs repeatedly, we will not mention the high probability bound in our theorems.
Revised: Since we will use the estimates in Lemma D.2 in the rest of our proofs repeatedly, we will not mention the high probability bound in our theorems.

Original: It should be understood that our theorems hold with the high probability bound given in Lemma D.2.
Revised: It should be understood that our theorems hold with the high probability bound given in Lemma D.2.

Original: Remark D.4 .
Revised: Remark D.4 .

Original: Recall that under Condition D.1, the term (cid:112) log( mKn ) =
Revised: Recall that under Condition D.1, the term (cid:112) log( mKn ) =

Original: polyloglog( d ) .
Revised: polyloglog( d ) .

Original: Its presence/absence will not affect any of our proofs.
Revised: Its pres-

Original: Lemma D.5.
Revised: Lemma D.5.

Original: Let { x t , y t } be two positive sequences updated as
Revised: Let { x t , y t } be two positive sequences updated as

Original: [Equation]
Revised: [Equation]

Original: where x 0 = o (1) , y 

Revised: where x 0 = o (1) , y

Original: = o (1) ,
Revised: = o (1) ,

Original: A =
Revised: A =

Original: o (1) , and B =
Revised: o (1) , and B =

Original: o (1) .
Revised: o (1) .

Original: For any D = O (1) , let T be the ﬁrst iteration that x t ≥ D .
Revised: For any D = O (1) , let T be the ﬁrst iteration that x t ≥ D .

Original: We have that y T ≤ O ( Gy 0 ) if x 0 ≥ O (cid:0) G − A B (cid:1) .
Revised: We have that y T ≤ O ( Gy 0 ) if x 0 ≥ O (cid:0) G − AB (cid:1) .

Original: Proof.
Revised: Proof.

Original: Let us consider the above two sequences x t and y t with the inequalities replaced with equality.
Revised: Let us consider the above two sequences x t and y t with the inequalities replaced with equality.

Original: Note that this will not affect our conclusion.
Revised: Note that this will not affect our conclusion.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: And for simplicity, let us also assume that x T = D .
Revised: And for simplicity, let us also assume that x T = D .

Original: Then we have Dx 0 =
Revised: Then we have Dx 0 =

Original: (1 + A ) T , which entails
Revised: (1 + A ) T , which entails

Original: [Equation]
Revised: [Equation]

Original: Let T (cid:48) be the ﬁrst iteration such that y T (cid:48) ≥ Gy 0 .
Revised: Let T (cid:48) be the ﬁrst iteration such that y T (cid:48) ≥ Gy 0 .

Original: Again, for simplicity, let us suppose that y T (cid:48) =
Revised: Again, for simplicity, let us suppose that y T (cid:48) =

Original: Gy 0 .
Revised: Gy 0 .

Original: Using the same procedure, we obtain that
Revised: Using the same procedure, we obtain that

Original: [Equation]
Revised: [Equation]

Original: Applying our assumption that x 0 ≥ O (cid:0) G − A B (cid:1) to Eq.
Revised: Applying our assumption that x 0 ≥ O (cid:0) G − AB (cid:1) to Eq.

Original: (D.1) and Eq.
Revised: (D.1) and Eq.

Original: (D.2)
Revised: (D.2)

Original: , we conclude that T <
Revised: , we conclude that T <

Original: T (cid:48) .
Revised: T (cid:48) .

Original: E T RAIN -T RAIN M ETHOD E. P HASE I
Revised: E T RAIN -T RAIN M ETHOD E. P HASE I

Original: In this section, we will show that under Condition D.1, our neural network will memorize the noise and will not learn the feature.
Revised: In this section, we will show that under Condition D.1, our neural network will memorize the noise and will not learn the feature.

Original: Let Ξ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) :
Revised: Let Ξ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) :

Original: j = y k,i , r ∈
Revised: j = y k,i , r ∈

Original: [ m ] (cid:111) and Λ ( t ) j = max (cid:110)
Revised: [ m ] (cid:111) and Λ ( t ) j = max (cid:110)

Original: j (cid:104) w ( t, 0) j,r , ν (cid:105) :
Revised: j (cid:104) w ( t, 0) j,r , ν (cid:105) :

Original: r ∈ [ m ] (cid:111) .
Revised: r ∈ [ m ] (cid:111) .

Original: Let T (1) k,i be the ﬁrst iteration such that Ξ ( t ) k,i ≥ m {+− 1 / 2 (cid:16) 1 + γσ 
 ξ d n (cid:17)+}
Revised: Let T (1) k,i be the ﬁrst iteration such that Ξ ( t ) k,i ≥ m {+− 1 / 2 (cid:16) 1 + γσ 
 ξ d n (cid:17)+}

Original: [Equation]
Revised: − J =

Original: − J = (cid:101) Θ(1) and let T (1) =
Revised: (cid:101) Θ(1) and let T (1) =

Original: max k,i T (1) k,i and (ˆ k, ˆ ı ) =
Revised: max k,i T (1) k,i and (ˆ k, ˆ ı ) =

Original: arg max k,i T (1) k,i .
Revised: arg max k,i T (1) k,i .

Original: So T (1) is the time for the slowest learned noise to have an inner product of size (cid:101) Θ(1) .
Revised: So T (1) is the time for the slowest learned noise to have an inner product of size (cid:101) Θ(1) .

Original: And this is witnessed by the noise vector ξ ˆ k, ˆ ı .
Revised: And this is witnessed by the noise vector ξ ˆ k, ˆ ı .

Original: Lemma E.1 (Restatement of Lemma 6.1) .
Revised: Lemma E.1 (Restatement of Lemma 6.1) .

Original: Under Condition D.1 and Condition 4.1, if one uses the train-train method, then for any t ≤ T (1)
Revised: Under Condition D.1 and Condition 4.1, if one uses the train-train method, then for any t ≤ T (1)

Original: For anyk ∈ [ K ] , r ∈ [ m ] and j ∈ {− 1 , 1 } we have [Equation] [Equation]
Revised: For any

Original: For j = y ˆ k, ˆ ı and r = arg max r (cid:48) ∈ [ m ] (cid:104)

Original: [Equation]

Original: ( t, 0) j,r [Equation] ˆ [Equation]
Revised: k ∈ [ K ] , r ∈

Original: [Equation]

Original: Proof.

Original: The proof for both parts relies on the following hypothesis which we will verify inductively later: [Equation] [Table] t
Revised: m

Original: [Table]

Original: τ [Table] j[Table]
Revised: and j

Original: r

Original: [Table]

Original: ν k [Table] = o [Table]
Revised: ∈ {− 1 ,

Original: ) , for all [Table] ∈ [ J ] , k ∈ [ K ] , r ∈ [ m ] , j ∈ {− 1 , 1 } .
Revised: } we have

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: , ν

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: For j = y ˆ k, ˆ ı and r = arg max

Original: ı
Revised: r

Original: [Table]

Original: ] if

Original: [Table]
Revised: [Equation]

Original: and
Revised: ∈

Original: [Table]
Revised: [ m ] [Equation] w ( t,

Original: [Table]
Revised: [Equation]

Revised: ,r , ξ ˆ

Original: [Table]
Revised: [Equation]

Revised: , ˆ

Original: (cid:105) >
Revised: [Equation] [Equation] Proof. The proof for both parts relies on the following hypothesis which we will verify inductively later:

Original: [Table]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁrst suppose that the above hypothesis hold.
Revised: Let us ﬁrst suppose that the above hypothesis hold.

Original: By Eq.
Revised: By Eq.

Original: (C.1)
Revised: (C.1)

Original: , we know j (cid:104) w ( t,τ ) j,r , ν k (cid:105) is an increasing sequence in τ .
Revised: , we know j (cid:104) w ( t,τ ) j,r , ν k (cid:105) is an increasing sequence in τ .

Original: Without loss of generality, suppose that j (cid:104) w ( t, 0) j,r , ν k (cid:105) > 0 .
Revised: Without loss of generality, suppose that j (cid:104) w ( t, 0) j,r , ν k (cid:105) > 0 .

Original: Then Hypothesis
Revised: Then Hypothesis

Original: E.3 implies
Revised: E.3 implies

Original: [Equation]
Revised: [Equation]

Original: Then using the fact that − (cid:96) (cid:48) ( · ) ≤ 1 , we can apply Eq.
Revised: Then using the fact that − (cid:96) (cid:48) ( · ) ≤ 1 , we can apply Eq.

Original: (C.1) repeatedly to get
Revised: (C.1) repeatedly to get

Original: [Equation]
Revised: [Equation]

Original: where we have also used (cid:107) ν k (cid:107) 

Revised: where we have also used (cid:107) ν k (cid:107)

Original: = (cid:107) ν (cid:107) 

Revised: = (cid:107) ν (cid:107)

Original: + (cid:107) z k (cid:107) 22 = 

Revised: + (cid:107) z k (cid:107) 22 =

Original: + o (1) .
Revised: + o (1) .

Original: This proves part (1) of the lemma.
Revised: This proves part (1) of the lemma.

Original: Let j =
Revised: Let j =

Original: y
Revised: y

Original: ˆ k, ˆ ı .
Revised: ˆ k, ˆ ı .

Original: Deﬁne r ( t ) =
Revised: Deﬁne r ( t ) =

Original: arg max r (cid:48) ∈ [ m ] (cid:104) w ( t, 0) j,r (cid:48) , ξ ˆ k, ˆ ı (cid:105) .
Revised: arg max r (cid:48) ∈ [ m ] (cid:104) w ( t, 0) j,r (cid:48) , ξ ˆ k, ˆ ı (cid:105) .

Original: By Eq.
Revised: By Eq.

Original: (C.2), we have (cid:104) w( t,
Revised: (C.2), we have (cid:104) w

Original: [Equation]
Revised: [Equation]

Original: + [Table] j,r ( t ) , ξ ˆ k, ˆ [Table]
Revised: t,τ

Original: (cid:105) ≥

Original: [Equation]
Revised: [Equation]

Original: w
Revised: j,

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table] [Table] [Table] =
Revised: t

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is because − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 .
Revised: where the ﬁrst inequality is because − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 .

Original: And we have used Lemma D.2 and
Revised: And we have used Lemma D.2 and

Original: Hypothesis E.2 to get the second inequality.
Revised: Hypothesis E.2 to get the second inequality.

Original: Let us compare the size of the second and third term on the right hand side of Eq.
Revised: Let us compare the size of the second and third term on the right hand side of Eq.

Original: (E.6).
Revised: (E.6).

Original: At ( t, τ ) =
Revised: At ( t, τ ) =

Original: (0 , 0) , if we leave out γ in both terms and consider, the third term is of size Θ( σ 0 σ 3 ξ d 3 / 2 /n )
Revised: (0 , 0) , if we leave out γ in both terms and consider, the third term is of size Θ( σ 0 σ 3 ξ d 3 / 2 /n )

Original: = Θ(log( d ) 1 . 5 / √ d ) , whereas the second term always has size O ( σ 2 ξ √ d (cid:112) log( Kn ))
Revised: = Θ(log( d ) 1 . 5 / √ d ) , whereas the second term always has size O ( σ 2 ξ √ d (cid:112) log( Kn ))

Original: =
Revised: =

Original: O (log( d ) / √ d ) O
Revised: O (log( d ) / √ d ) O

Original: (polyloglog( d )) .
Revised: (polyloglog( d )) .

Original: Hence the third term will dominate the second term at ( t, τ ) =
Revised: Hence the third term will dominate the second term at ( t, τ ) =

Original: Note that the size of the second term does not change as t and τ increase, meaning that if at time ( t ≥ 0 , τ ≥ 0) we have
Revised: Note that the size of the second term does not change as t and τ increase, meaning that if at time ( t ≥ 0 , τ ≥ 0) we have

Original: [Equation]
Revised: [Equation]

Original: then it always holds that the third term dominates the second term on the right hand side of Eq.
Revised: then it always holds that the third term dominates the second term on the right hand side of Eq.

Original: (E.6), which implies
Revised: (E.6), which implies

Original: [Equation]
Revised: [Equation]

Original: Applying Eq.
Revised: Applying Eq.

Original: (E.7)
Revised: (E.7)

Original: J times at t = 0 and using Hypothesis E.4, we obtain
Revised: J times at t = 0 and using Hypothesis E.4, we obtain

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (C.6), we have
Revised: (C.6), we have

Original: [Equation]
Revised: [Equation]

Original: where we have used Hypothesis E.2 and the fact that − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 to get ﬁrst inequality.
Revised: where we have used Hypothesis E [Equation] 2 and the fact that − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 to get ﬁrst inequality.

Original: Let us compare the second and third term on the right hand side of Eq.
Revised: Let us compare the second and third term on the right hand side of Eq.

Original: (E.9).
Revised: (E.9).

Original: At t = 0 , using
Revised: At t = 0 , using

Original: Eq.
Revised: Eq.

Original: (E.8) we have that
Revised: (E.8) we have that

Original: [Equation]
Revised: [Equation]

Original:  If we drop η in both terms, under Condition D.1 and Condition 4.1, one can calculate that the third term has size Θ(log( d ) 8 / √ d ) whereas the second term has size O (log( d ) / √ d )
Revised: (E.10) If we drop η in both terms, under Condition D.1 and Condition 4.1, one can calculate that the third term has size Θ(log( d ) 8 / √ d ) whereas the second term has size O (log( d ) / √ d )

Original: O (polyloglog( d )) .
Revised: O (polyloglog( d )) .

Original: Thus the third term dominates the second term in Eq.
Revised: Thus the third term dominates the second term in Eq.

Original: (E.9) at t = 0 .
Revised: (E.9) at t = 0 .

Original: And we get 
Revised: And we get (cid:104) w (1 , 0) j [Equation] r (1)

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality in the ﬁrst line is by deﬁnition of r ( t ) .
Revised: where the ﬁrst inequality in the ﬁrst line is by deﬁnition of r ( t ) .

Original: We note that the size of the second term on the right hand side of Eq.
Revised: We note that the size of the second term on the right hand side of Eq.

Original: (E.9) does not change as t and τ increase.
Revised: (E.9) does not change as t and τ increase.

Original: Thus, if (cid:104) w ( t, 0) j,r , ξ ˆ k, ˆ ı (cid:105) = Ω(1) Ξ (0)ˆ k, ˆ ı for some t > 0 , Eq.
Revised: Thus, if (cid:104) w ( t, 0) j,r , ξ ˆ k, ˆ ı (cid:105) = Ω(1) Ξ (0)ˆ k, ˆ ı for some t > 0 , Eq.

Original: (E.10) implies that σ (cid:48) ( (cid:104) w ( t,J ) j,r , ξ ˆ k, ˆ ı (cid:105) )
Revised: (E.10) implies that σ (cid:48) ( (cid:104) w ( t,J ) j,r , ξ ˆ k, ˆ ı (cid:105) )

Original: =
Revised: =

Original: Ω(1) σ (cid:48) ( (cid:104) w (0 ,J ) j,r , ξ ˆ k, ˆ ı (cid:105) ) .
Revised: Ω(1) σ (cid:48) ( (cid:104) w (0 ,J ) j,r , ξ ˆ k, ˆ ı (cid:105) ) .

Original: Then we have that the third term will dominate the second term on the right hand side of Eq.
Revised: Then we have that the third term will dominate the second term on the right hand side of Eq.

Original: (E.9) at t > 0 , which yields (cid:104) w ( t +1 , 0) j,r ( t ) , ξ ˆ k, ˆı (cid:105) >
Revised: (E.9) at t > 0 , which yields (cid:104) w ( t +1 , 0) j,r ( t ) , ξ ˆ k, ˆ

Revised: [Equation]

Original:  (cid:104) w ( t, 0) j,r ( t ) , ξ ˆ k, ˆ ı (cid:105) since the third term on the right hand side of
Revised: (cid:105) > (cid:104) w ( t, 0) j,r ( t ) , ξ ˆ k, ˆ ı (cid:105) since the third term on the right hand side of

Original: Eq.
Revised: Eq.

Original: (E.9) is positive.
Revised: (E.9) is positive.

Original: We have shown that
Revised: We have shown that

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]

Original: Then using the same derivation of Eq.
Revised: Then using the same derivation of Eq.

Original: (E.8), we obtain
Revised: (E.8), we obtain

Original: [Equation]
Revised: [Equation]

Original: for any t ≤ T (1) .
Revised: for any t ≤ T (1) .

Original: This proves the second part of the lemma.
Revised: This proves the second part of the lemma.

Original: We have the following theorem characterizing the size of Λ T (1) j .
Revised: We have the following theorem characterizing the size of Λ T (1) j .

Original: Theorem E.2.
Revised: Theorem E.2.

Original: Under Condition D.1 and Condition 4.1, if one uses the train-train method, then for any t ≤ T (1)
Revised: Under Condition D.1 and Condition 4.1, if one uses the train-train method, then for any t ≤ T (1)

Original: Λ tj = Θ(1) Λ 0 j = (cid:101) O
Revised: Λ tj = Θ(1) Λ 0 j = (cid:101) O

Original: ( d − 1 / 2 ) , for j ∈ {− 1 , 1 } . 2.
Revised: ( d − 1 / 2 ) , for j ∈ {− 1 , 1 } . 2.

Original: For any k ∈ [ K ] , i ∈ [ n ] , j = − y k,i and r ∈ [ m ] , we have
Revised: For any k ∈ [ K ] , i ∈ [ n ] , j = − y k,i and r ∈ [ m ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that the outer-loop for the feature is given by
Revised: Recall that the outer-loop for the feature is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where the summation is over all k (
Revised: where the summation is over all k (

Original: cid:48) ∈
Revised: cid:48) ∈

Original: [ K ] and i ∈ [ n ] .
Revised: [ K ] and i ∈ [ n ] .

Original: Since (cid:104) ν k , ν k (cid:48) (cid:105) = (cid:107) ν (cid:107) 2 + (cid:104)
Revised: Since (cid:104) ν k , ν k (cid:48) (cid:105) = (cid:107) ν (cid:107) 2 + (cid:104)

Original: z k , z k (cid:48) (cid:105) ≥ 1 − o (1) > 0 , we get that
Revised: z k , z k (cid:48) (cid:105) ≥ 1 − o (1) > 0 , we get that

Original: j (cid:104) w ( t, 0) j,r , ν k (cid:105) is an increasing sequence in t for any k ∈
Revised: j (cid:104) w ( t, 0) j,r , ν k (cid:105) is an increasing sequence in t for any k ∈

Original: [ K ] and r ∈ [ m ] .
Revised: [ K ] and r ∈ [ m ] .

Original: And Eq.
Revised: And Eq.

Original: (C.8) shows that j (cid:104) w ( t, 0) j,r , ν (cid:105) is an increasing sequence in t .
Revised: (C.8) shows that j (cid:104) w ( t, 0) j,r , ν (cid:105) is an increasing sequence in t .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where is ﬁrst inequality is by Hypothesis E.3 and our assumption that
Revised: where is ﬁrst inequality is by Hypothesis E.3 and our assumption that

Original: j (cid:104) w ( t, 0) j,r , ν k (cid:105) > 0 .
Revised: j (cid:104) w ( t, 0) j,r , ν k (cid:105) > 0 .

Original: The second inequality is by Lemma E.1.
Revised: The second inequality is by Lemma E.1.

Original: And the last equality is due to our Hypothesis E.1.
Revised: And the last equality is due to our Hypothesis E.1.

Original: Without loss of generality, consider j = 1 .
Revised: Without loss of generality, consider j = 1 .

Original: Let r ( t ) =
Revised: Let r ( t ) =

Original: arg max r (cid:48) ∈ [ m ] (cid:104) w ( t, 0) 1 ,r (cid:48) , ν (cid:105) .
Revised: arg max r (cid:48) ∈ [ m ] (cid:104) w ( t, 0) 1 ,r (cid:48) , ν (cid:105) .

Original: We can upper bound the growth of the feature in the outer-loop by
Revised: We can upper bound the growth of the feature in the outer-loop by

Original: [Equation]
Revised: [Equation]

Original: where we have also used the fact that − (cid:96) (cid:48) ( · ) < 1 by deﬁnition.
Revised: where we have also used the fact that − (cid:96) (cid:48) ( · ) < 1 by deﬁnition.

Original: Note that Eq.
Revised: Note that Eq.

Original: (E.12) also holds for j = − 1 , i.e., Λ ( t +1)−≤ Λ [Equation] ) −
Revised: (E.12) also holds for j = − 1 , i.e., Λ ( t +1)

Original: [Equation] 1 + O [Equation] 1) ηγ J (cid:1) using the same derivation.
Revised: −≤ Λ ( t ) − 1 (cid:0) 1 + O (1) ηγ J (cid:1) using the same derivation.

Original: Next, we want to ﬁnd a lower bound on the learning speed of the noise.
Revised: Next, we want to ﬁnd a lower bound on the learning speed of the noise.

Original: After that, we wish to show that the lower bound for the noise actually grows much faster than the upper bound of the memorization of the feature, which implies that noise will be learnt sufﬁciently well before our neural network can pick up any learning on the feature.
Revised: After that, we wish to show that the lower bound for the noise actually grows much faster than the upper bound of the memorization of the feature, which implies that noise will be learnt sufﬁciently well before our neural network can pick up any learning on the feature.

Original: Consider j = y ˆ k, ˆ ı .
Revised: Consider j = y ˆ k, ˆ ı .

Original: Denote r (cid:48) ( t 
Revised: Denote r (cid:48) ( t ) =

Revised: arg max r ∈ [ m ] (cid:104) w

Original: [Equation]
Revised: [Equation]

Original: arg max r ∈
Revised: j

Original: [Equation]
Revised: [Equation]

Original: m
Revised: ξ

Original: [Equation]
Revised: [Equation]

Original: Plugging Lemma E. and Hypothesis E.4 into Eq.
Revised: Plugging Lemma E. and Hypothesis E.4 into Eq.

Original: (E.9)
Revised: (E.9)

Original: we have the following upper bound hold for all t ≤ T (1)
Revised: we have the following upper bound hold for all t ≤ T (1)

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by deﬁnition of r (cid:48) ( t ) .
Revised: where the ﬁrst inequality is by deﬁnition of r (cid:48) ( t ) .

Original: Now we can compare the learning speed of the feature and noise:
Revised: Now we can compare the learning speed of the feature and noise:

Original: [Equation]
Revised: [Equation]

Original: We can apply Lemma D.5 once we check all of its conditions are satisﬁed.
Revised: We can apply Lemma D.5 once we check all of its conditions are satisﬁed.

Original: We have Λ (0)
Revised: We have Λ (0)

Original: j = (cid:101) Θ( d − 1 / 2 ) =
Revised: j = (cid:101) Θ( d − 1 / 2 ) =

Original: o (1) , Ξ (0)ˆ k, ˆ ı = (cid:101)
Revised: o (1) , Ξ (0)ˆ k, ˆ ı = (cid:101)

Original: Θ( d − 1 / 2 ) =
Revised: Θ( d − 1 / 2 ) =

Original: o (1) .
Revised: o (1) .

Original: Let G = 2 ,
Revised: Let G = 2 ,

Original: A
Revised: A

Original: = O (1) ηγ J and B = Ω(1)
Revised: = O (1) ηγ J and B = Ω(1)

Original: ηγ J K (cid:0) σ 2 ξ d n (cid:1)
Revised: ηγ J K (cid:0) σ 2 ξ d n (cid:1)

Original: J +1 , we have Ξ (0)ˆ k, ˆ ı = (cid:101)
Revised: J +1 , we have Ξ (0)ˆ k, ˆ ı = (cid:101)

Original: Θ( d − 1 / 2 ) >
Revised: Θ( d − 1 / 2 ) >

Original: G − A/B = 2 − log( d ) 5 . 5 .
Revised: G − A/B = 2 − log( d ) 5 . 5 .

Original: Therefore, by Lemma
Revised: Therefore, by Lemma

Original: D.5, for any t ≤ T (1) , we have Λ ( t ) j ≤ 2 Λ (0) j .
Revised: D.5, for any t ≤ T (1) , we have Λ ( t ) j ≤ 2 Λ (0) j .

Original: This proves part (1) of our theorem.
Revised: This proves part (1) of our theorem.

Original: We now prove part (2) of our theorem.
Revised: We now prove part (2) of our theorem.

Original: Let k ∈
Revised: Let k ∈

Original: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j = − y k,i .
Revised: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j = − y k,i .

Original: By Eq.
Revised: By Eq.

Original: (C.6) 
Revised: (C.6) (cid:104) w ( t +1 , 0) j,r

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because both − (cid:96) (cid:48) ( · ) and σ (cid:48) ( · ) are bounded by 1.
Revised: where the second inequality is because both − (cid:96) (cid:48) ( · ) and σ (cid:48) ( · ) are bounded by 1.

Original: Using Eq.
Revised: Using Eq.

Original: (E.15) and the deﬁnition of T (1) , we can calculate that T (1) =
Revised: (E.15) and the deﬁnition of T (1) , we can calculate that T (1) =

Original: (cid:101) O
Revised: (cid:101) O

Original: (1) under Condition D.1.
Revised: (1) under Condition D.1.

Original: Therefore, for any t ≤ T (1) − 1 we have
Revised: Therefore, for any t ≤ T (1) − 1 we have

Original: [Equation]
Revised: [Equation]

Original: where we have used Lemma D.2 to get the second inequality.
Revised: where we have used Lemma D.2 to get the second inequality.

Original: It remains to verify Hypothesis E.1-E.4.
Revised: It remains to verify Hypothesis E.1-E.4.

Original: Let us suppose that Hypothesis E.1-E.4 hold for all t <
Revised: Let us suppose that Hypothesis E.1-E.4 hold for all t <

Original: T (1) .
Revised: T (1) .

Original: Then we have Λ ( t +1)
Revised: Then we have Λ ( t +1)

Original: j ≤ 2 Λ (0) j , and (cid:104) w ( t +1 , 0) j,r , ξ k,i (cid:105) ≤ (cid:101) O
Revised: j ≤ 2 Λ (0) j , and (cid:104) [Equation] [Equation] d

Original: ( d − 1 / 2 ) for any k ∈ [ K ] , i ∈ [ n ] , r ∈ [ m ] and j = − y k,i by Theorem E.2.
Revised: [Equation] 1 / 2 ) for any k ∈ [ K ] , i ∈ [ n ] , r ∈ [ m ] and j = − y k,i by Theorem E.2.

Original: Proof of Hypothesis E.1.
Revised: Proof of Hypothesis E.1.

Original: For each task-speciﬁc feature z k , we have at t = 0 using Lemma D.
Revised: For each task-speciﬁc feature z k , we have at t = 0 using Lemma D.

Original: [Equation]
Revised: [Equation]

Original: Since σ s √ d
Revised: Since σ s √ d

Original: = o (1) , we see that Hypothesis E.1 holds at t = 0 .
Revised: = o (1) , we see that Hypothesis E.1 holds at t = 0 .

Original: We want to prove that it holds at t + 1 .
Revised: We want to prove that it holds at t + 1 .

Original: For any k ∈ [ K ] , r ∈ [ m ] we have
Revised: For any k ∈ [ K ] , r ∈ [ m ] we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: where we used Condition 4.1 to get the ﬁrst inequality.
Revised: where we used Condition 4.1 to get the ﬁrst inequality.

Original: The last inequality is because log( d ) (cid:28) d .
Revised: The last inequality is because log( d ) (cid:28) d .

Original: Suppose that at time t + 1 , there exists some ˜ r ∈ [ m ] and ˜ k ∈
Revised: Suppose that at time t + 1 , there exists some ˜ r ∈ [ m ] and ˜ k ∈

Original: [ K ] such that
Revised: [ K ] such that

Original: [Equation]
Revised: [Equation]

Original: Then, by Eq.
Revised: Then, by Eq.

Original: (E.17) we get −η
Revised: (E.17) we get −

Original: [Equation]
Revised: [Equation]

Original: Here the second inequality is by Lemma D
Revised: Kn

Original: [Equation]
Revised: [Equation]

Revised: Here the second inequality is by Lemma D.2.

Original: But then, Eq.
Revised: But then, Eq.

Original: (C
Revised: (C.8) implies

Original: [Equation]
Revised: [Equation]

Original: 8) implies
Revised: We used Lemma D

Original: [Equation]
Revised: [Equation]

Original: We used Lemma D.2 to get the third line and last line.
Revised: 2 to get the third line and last line.

Original: This is a contradiction, since we should have
Revised: This is a contradiction, since we should have

Original: [Equation]
Revised: [Equation]

Original: Therefore for all k ∈ [ K ] we conclude
Revised: Therefore for all k ∈ [ K ] we conclude

Original: [Equation]
Revised: [Equation]

Original: This proves that Hypothesis E.1 holds at time t + 1 .
Revised: This proves that Hypothesis E [Equation] 1 holds at time t + 1 .

Original: Proof of Hypothesis E.2.
Revised: Proof of Hypothesis E.2.

Original: Without loss of generality, let us assume that y ˆ k, ˆ ı = 1 .
Revised: Without loss of generality, let us assume that y ˆ k, ˆ ı = 1 .

Original: By our estimates in Lemma D.2, Hypothesis E.2 clearly holds at initialization.
Revised: By our estimates in Lemma D [Equation] 2, Hypothesis E.2 clearly holds at initialization.

Original: Suppose it holds at time t .
Revised: Suppose it holds at time t .

Original: Recall
Revised: Recall

Original: Eq.
Revised: Eq.

Original: (C.7) and that (cid:88)
Revised: (C.7) and that [Equation] (cid:105) ) (cid:1)

Original: [Equation]
Revised: [Equation]

Original: ) , (E.18) by the non-negativity of σ ( · ) .
Revised: by the non-negativity of σ ( · ) .

Original: For any r ∈ [ m ] , we have
Revised: For any r ∈ [ m ] , we have

Original: [Equation]
Revised: [Equation]

Original: where we used Hypothesis E
Revised: where we used Hypothesis E.1 to get the third line. Then, by Eq.

Original: [Equation]
Revised: (E.5)

Original: 1 to get the third line
Revised: we have that for any τ ∈ [ J ]

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: t

Original: [Equation]
Revised: [Equation]

Original: by Eq.

Original: (E

Original: [Equation]
Revised: [Equation]

Original: 5) we have that for any τ∈ [ J ]
Revised: τ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: We also have that
Revised: We also have that

Original: [Equation]
Revised: [Equation]

Original: where the inequality is due to t + 1 < T (1) which implies Ξ ( t +1)
Revised: where the inequality is due to t + 1 < T (1) which implies Ξ ( t +1)

Original: ˆ k, ˆ ı < m − 1 / 2 (cid:16) 1 + γσ 

Revised: ˆ k, ˆ ı < m − 1 / 2 (cid:16) 1 + γσ

Original: ξ d n (cid:17) − J .
Revised: ξ d n (cid:17) − J .

Original: Hence, by Eq.
Revised: Hence, by Eq.

Original: (E.18) we get that − (cid:96) (cid:48)
Revised: (E.18) we get that − (cid:96) (cid:48)

Original: ( t +1 , 0)
Revised: ( t +1 , 0)

Original: ˆ k, ˆ ı = − (cid:96) (cid:48)
Revised: ˆ k, ˆ ı = − (cid:96) (cid:48)

Original: ( o (1)) = Ω(1) .
Revised: ( o (1)) = Ω(1) .

Original: Now, suppose that − (cid:96) (cid:48) ( t +1 ,τ ) ˆ k, ˆ ı = Ω(1) for some 0 ≤ τ ≤ J − 1 .
Revised: Now, suppose that − (cid:96) (cid:48) ( t +1 ,τ ) ˆ k, ˆ ı = Ω(1) for some 0 ≤ τ ≤ J − 1 .

Original: We wish to show that − (cid:96) (cid:48)
Revised: We wish to show that − (cid:96) (cid:48)

Original: ( t +1 ,τ +1)
Revised: ( t +1 ,τ +1)

Original: ˆ k, ˆ ı = Ω(1) .
Revised: ˆ k, ˆ ı = Ω(1) .

Original: By our previous argument, it sufﬁces to show (cid:80)
Revised: By our previous argument, it sufﬁces to show (cid:80)

Original: mr =1 σ ( (cid:104) w ( t +1 ,τ +1) 1 ,r , ξ ˆ k, ˆ ı (cid:105) )
Revised: mr =1 σ ( (cid:104) w ( t +1 ,τ +1) 1 ,r , ξ ˆ k, ˆ ı (cid:105) )

Original: =
Revised: =

Original: O (1) .
Revised: O (1) .

Original: Because that would imply − (cid:96) (cid:48) ( t +1 ,τ +1)
Revised: Because that would imply − (cid:96) (cid:48) ( t +1 ,τ +1)

Original: ˆ k, ˆ ı = − (cid:96) (cid:48)
Revised: ˆ k, ˆ ı = − (cid:96) (cid:48)

Original: ( O (1)) = Ω(1) .
Revised: ( O (1)) = Ω(1) .

Original: We have,m (cid:88) r =
Revised: We have,

Original: [Equation]
Revised: [Equation]

Original: n
Revised: ≤ O (

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because τ + 1 ≤ J .
Revised: where the second inequality is because τ + 1 ≤ J .

Original: This implies − (cid:96) (cid:48)
Revised: This implies − (cid:96) (cid:48)

Original: ( t +1 ,τ +1)
Revised: ( t +1 ,τ +1)

Original: ˆ k, ˆ ı
Revised: ˆ k, ˆ ı

Original: [Equation]
Revised: [Equation]

Original: O

Original: [Equation]
Revised: [Equation]

Original: = Ω
Revised: = Ω

Original: [Equation]
Revised: [Equation]

Original: 1) and we are done.
Revised: 1) and we are done.

Original: Proof of Hypothesis E.3.
Revised: Proof of Hypothesis E.3.

Original: By our estimates in Lemma D.2, Hypothesis E.3 clearly holds at initialization.
Revised: By our estimates in Lemma D.2, Hypothesis E.3 clearly holds at initialization.

Original: Suppose it holds at time t .
Revised: Suppose it holds at time t .

Original: Using Eq.
Revised: Using Eq.

Original: (E.19)
Revised: (E.19)

Original: , we immediately have that for any τ ∈
Revised: , we immediately have that for any τ ∈

Original: [ J ] , r ∈ [ m ] and k ∈ [ K ] ,
Revised: [ J ] , r ∈ [ m ] and k ∈ [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Hypothesis E.4.
Revised: Proof of Hypothesis E.4.

Original: By our estimates in Lemma D.2, Hypothesis E.4 clearly holds at initialization.
Revised: By our estimates in Lemma D.2, Hypothesis E.4 clearly holds at initialization.

Original: Suppose it holds at time t .
Revised: Suppose it holds at time t .

Original: Suppose that (cid:104) w ( t +1 , 0) j,r , ξ ˆ k, ˆ ı (cid:105) > 0 for j = y ˆ k, ˆ ı and some r ∈ [ m ] .
Revised: Suppose that (cid:104) w ( t +1 , 0) j,r , ξ ˆ k, ˆ ı (cid:105) > 0 for j = y ˆ k, ˆ ı and some r ∈ [ m ] .

Original: Recall that by the assumption of t + 1 ≤ T (1) , we have[Equation] n
Revised: Recall that by the assumption of t + 1 ≤ T (1) , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, by deﬁnition of σ (cid:48) ( · ) we have σ (cid:48)
Revised: Hence, by deﬁnition of σ (cid:48) ( · ) we have σ (cid:48)

Original: ( (cid:104) w ( t +1 , 0) j,r , ξ ˆ k, ˆ ı (cid:105) )
Revised: ( (cid:104) w ( t +1 , 0) j,r , ξ ˆ k, ˆ ı (cid:105) )

Original: = 2 (cid:104) w ( t +1 , 0)
Revised: = 2 (cid:104) w ( t +1 , 0)

Original: j,r , ξ ˆ k,i (cid:105) .
Revised: j,r , ξ ˆ k,i (cid:105) .

Original: While (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) =
Revised: While (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) =

Original: o (1) , we have σ (cid:48)
Revised: o (1) , we have σ (cid:48)

Original: ( (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) )
Revised: ( (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) )

Original: = 2 (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) .
Revised: = 2 (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) .

Original: By Eq.
Revised: By Eq.

Original: (C.2) we have (cid:104) w ( t +1 ,τ +1)
Revised: (C.2) we have (cid:104) w ( t +1 ,τ +1)

Original: j,r , ξ ˆ k, ˆ ı (cid:105) ≤ (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) + γ n (cid:88) i (cid:54)
Revised: j,r , ξ ˆ k, ˆ ı (cid:105) ≤ (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) + γ n (cid:88) i (cid:54)

Original: =
Revised: =

Original: ˆ ı
Revised: ˆ ı

Original: O (1) |(cid:104)
Revised: O (1) |(cid:104)

Original: ξ
Revised: ξ

Original: [Table]
Revised: [Table]

Original: ,i , ξ ˆ
Revised: ,i , ξ ˆ

Original: [Table]
Revised: [Table]

Original: , ˆ ı (cid:105)| − γ n(cid:96) (cid:48) (
Revised: , ˆ ı (cid:105)| − γ n(cid:96) (cid:48) (

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ı σ (cid:48) (
Revised: ı σ (cid:48) (

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: , ξ
Revised: , ξ

Original: [Table]
Revised: [Table]

Original: , ˆ
Revised: , ˆ

Original: [Table]
Revised: [Table]

Original: (cid:105) )
Revised: (cid:105) )

Original: (cid:13)(cid:13)(cid:13) ξ
Revised: (cid:13)(cid:13)(cid:13) ξ

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ı (cid:13)(cid:13)(cid:13)  = (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) + γ O (1) O ( σ 2 ξ √ d (cid:112) log( Kn ))
Revised: ı (cid:13)(cid:13)(cid:13)  = (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) + γ O (1) O ( σ 2 ξ √ d (cid:112) log( Kn ))

Original: + γ n O (1) σ (cid:48)
Revised: + γ n O (1) σ (cid:48)

Original: ( (cid:104) w (
Revised: ( (cid:104) w (

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: We used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 to get the ﬁrst line.
Revised: We used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 to get the ﬁrst line.

Original: We used Lemma D.2 for the second line.
Revised: We used Lemma D.2 for the second line.

Original: It is straightforward to check recursively that (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) ≤
Revised: It is straightforward to check recursively that (cid:104) w ( t +1 ,τ ) j,r , ξ ˆ k, ˆ ı (cid:105) ≤

Original: o (1) for τ ∈
Revised: o (1) for τ ∈

Original: [ J − 1] and hence Eq.
Revised: [ J − 1] and hence Eq.

Original: (E.20) holds for τ ∈
Revised: (E.20) holds for τ ∈

Original: [ J − 1] .
Revised: [ J − 1] .

Original: In the meantime, this also implies σ (cid:48)
Revised: In the meantime, this also implies σ (cid:48)

Original: ( (cid:104)
Revised: ( (cid:104)

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: t +1 ,τ
Revised: t +1 ,τ

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: ( t +
Revised: ( t +

Original: [Equation]
Revised: [Equation]

Original: , ξ
Revised: , ξ

Original: [Equation]
Revised: [Equation]

Original: k,
Revised: k,

Original: [Equation]
Revised: [Equation]

Original: ı (cid:105) for τ ∈ [ J − 1] .
Revised: ı (cid:105) for τ ∈ [ J − 1] .

Original: Then Eq.
Revised: Then Eq.

Original: (E.20) implies
Revised: (E.20) implies

Original: [Equation]
Revised: [Equation]

Original: Therefore, σ (cid:48)
Revised: Therefore, σ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We note that by our deﬁnition, T (1) is the time taken for the slowest-learnt noise to be memorized by our neural network.
Revised: We note that by our deﬁnition, T (1) is the time taken for the slowest-learnt noise to be memorized by our neural network.

Original: Other noises are learned faster.
Revised: Other noises are learned faster.

Original: Our next theorem guarantees that other noises has almost the same inner product as the slowest-learnt noise at time T (1) .
Revised: Our next theorem guarantees that other noises has almost the same inner product as the slowest-learnt noise at time T (1) .

Original: Theorem E.3.
Revised: Theorem E.3.

Original: Under Condition D.1 and Condition 4.1, suppose one uses the train-train method.
Revised: Under Condition D.1 and Condition 4.1, suppose one uses the train-train method.

Original: For any k ∈ [ K ] , i ∈ [ n ] with ( k, i ) (cid:54) =
Revised: For any k ∈ [ K ] , i ∈ [ n ] with ( k, i ) (cid:54) =

Original: (ˆ k, ˆ ı ) , let
Revised: (ˆ k, ˆ ı ) , let

Original: t ∈
Revised: t ∈

Original: [ T (1) k,i , T (1) ] , then we have Ξ ( t ) k,i = (cid:101) Θ(1) .
Revised: [ T (1) k,i , T (1) ] , then we have Ξ ( t ) k,i = (cid:101) Θ(1) .

Original: More precisely, it holds that
Revised: More precisely, it holds that

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that Ξ ( t ) k,i ≥ Ω (cid:16) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J (cid:17) =
Revised: We ﬁrst show that Ξ ( t ) k,i ≥ Ω (cid:16) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J (cid:17) =

Original: (cid:101) Ω(1) .
Revised: (cid:101) Ω(1) .

Original: By deﬁnition of T (1) k,i , we know that Ξ ( t ) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J when t = T (1) k,i .
Revised: By deﬁnition of T (1) k,i , we know that Ξ ( t ) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J when t = T (1) k,i .

Original: Suppose there exists t ∈ [ T (1) k,i , T (1) ] such that it is the ﬁrst iteration such that Ξ ( t ) k,i < m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J .
Revised: Suppose there exists t ∈ [ T (1) k,i , T (1) ] such that it is the ﬁrst iteration such that Ξ ( t ) k,i < m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J .

Original: If such t does not exist, then we automatically have Ξ ( t ) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J for all t ∈
Revised: If such t does not exist, then we automatically have Ξ ( t ) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J for all t ∈

Original: [ T (1) k,i , T (1) ] .
Revised: [ T (1) k,i , T (1) ] .

Original: By deﬁnition of t we have Ξ ( t − 1) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J .
Revised: By deﬁnition of t we have Ξ ( t − 1) k,i ≥ m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − J .

Original: By Eq.
Revised: By Eq.

Original: (E.9), we have the following lower bound on Ξ ( t ) k,
Revised: (E.9), we have the following lower bound on Ξ ( t ) k,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst inequality we used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2.
Revised: where in the ﬁrst inequality we used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2.

Original: The second and the third line are direct calculations under Condition D.1.
Revised: The second and the third line are direct calculations under Condition D.1.

Original: Using the same proof as Hypothesis E.2, we have that − (cid:96) (cid:48) ( t,τ ) k,i = Ω(1) for any τ ∈ [ J ] .
Revised: Using the same proof as Hypothesis E.2, we have that − (cid:96) (cid:48) ( t,τ ) k,i = Ω(1) for any τ ∈ [ J ] .

Original: Using a similar derivation of Eq.
Revised: Using a similar derivation of Eq.

Original: (E.13)
Revised: (E.13)

Original: , we found that
Revised: , we found that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because Ξ ( t − 1) k,i
Revised: where the last inequality is because Ξ ( t − 1) k,i

Original: [Equation]
Revised: [Equation]

Original: ) , and (cid:0) 1+Ω(1) ησ 2 ξ d Kn (cid:0) γσ
Revised: ) , and (cid:0) 1+Ω(1) ησ 2 ξ d Kn (cid:0) γσ

Original: [Equation]
Revised: [Equation]

Original: d n (cid:1) J (cid:1) = Ω(polylog
Revised: d n (cid:1) J (cid:1) = Ω(polylog

Original: ( d ))
Revised: ( d ))

Original: (cid:29) O (1) .
Revised: (cid:29) O (1) .

Original: This shows that even if Ξ ( t ) k,i < Ξ ( t − 1) k,i at some t , we still have Ξ ( t ) k,i = Ω (cid:0) m − 1 / 2 (cid:0) 1+ γσ 2 ξ d n (cid:1) − J (cid:1) =
Revised: This shows that even if Ξ ( t ) k,i < Ξ ( t − 1) k,i at some t , we still have Ξ ( t ) k,i = Ω (cid:0) m − 1 / 2 (cid:0) 1+ γσ 2 ξ d n (cid:1) − J (cid:1) =

Original: (cid:101) Ω(1) and Ξ ( t +1) k,i > Ξ ( t − 1) k,i .
Revised: (cid:101) Ω(1) and Ξ ( t +1) k,i > Ξ ( t − 1) k,i .

Original: Hence, Ξ ( t ) k,i ≥ (cid:101) Ω(1) for all t ∈
Revised: Hence, Ξ ( t ) k,i ≥ (cid:101) Ω(1) for all t ∈

Original: [ T (1) k,i , T (1) ] .
Revised: [ T (1) k,i , T (1) ] .

Original: Next, we need to show that Ξ ( t ) k,i ≤ O (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +0 .
Revised: Next, we need to show that Ξ ( t ) k,i ≤ O (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +0 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: Let t (cid:48)
Revised: Let t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: where the inf of an empty set is deﬁned as ∞ .
Revised: where the inf of an empty set is deﬁned as ∞ .

Original: If t (cid:48)
Revised: If t (cid:48)

Original: = ∞ then we are done, because that implies Ξ ( T (1) )
Revised: = ∞ then we are done, because that implies Ξ ( T (1) )

Original: k,i <
Revised: k,i <

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assume t (cid:48) is ﬁnite.
Revised: Assume t (cid:48) is ﬁnite.

Original: We ﬁrst need to show that Ξ ( t (cid:48) )
Revised: We ﬁrst need to show that Ξ ( t (cid:48) )

Original: k,i =
Revised: k,i =

Original: O (cid:0) m
Revised: O (cid:0) m

Original: [Equation]
Revised: [Equation]

Original: / 2 (cid:0)
Revised: / 2 (cid:0)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 2 ξ d n (cid:1) − J +
Revised: 2 ξ d n (cid:1) − J +

Original: [Equation]
Revised: [Equation]

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: By deﬁnition of t (cid:48) , we know Ξ ( t
Revised: By deﬁnition of t (cid:48) , we know Ξ ( t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By Eq.
Revised: By Eq.

Original: (C.6), we have
Revised: (C.6), we have

Original: [Equation]
Revised: [Equation]

Original: where we used (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.
Revised: where we used (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.

Original: The last inequality is because (cid:101) O
Revised: The last inequality is because (cid:101) O

Original: ( d − 1 / 2 ) + Θ (cid:0) ησ 2 ξ d Kn (cid:1)
Revised: ( d − 1 / 2 ) + Θ (cid:0) ησ 2 ξ d Kn (cid:1)

Original: =
Revised: =

Original: o (1) m − 1 / 2 (cid:0) 1 + γσ 

Revised: o (1) m − 1 / 2 (cid:0) 1 + γσ

Original: ξ d n (cid:1) − J +0 .
Revised: ξ d n (cid:1) − J +0 .

Original: 5 by a direct calculation.
Revised: 5 by a direct calculation.

Original: We now show that − (cid:96) (cid:48)
Revised: We now show that − (cid:96) (cid:48)

Original: ( t,J ) k,
Revised: ( t,J ) k,

Original: i =
Revised: i =

Original: O (exp( − log( d ) 0 . 6 )) for all t ∈ [ t (cid:48) , T (1) ] .
Revised: O (exp( − log( d ) 0 . 6 )) for all t ∈ [ t (cid:48) , T (1) ] .

Original: Without loss of generality, assume y k,i = 1 .
Revised: Without loss of generality, assume y k,i = 1 .

Original: Let r ( t ) =
Revised: Let r ( t ) =

Original: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .
Revised: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .

Original: Using similar proof of Hypothesis E.2, we have that − (cid:96) (cid:48) ( t,τ ) k,i = Θ(1) for τ ∈ [ J − 1] .
Revised: Using similar proof of Hypothesis E.2, we have that − (cid:96) (cid:48) ( t,τ ) k,i = Θ(1) for τ ∈ [ J − 1] .

Original: Since (cid:104) w ( t, 0) 1 ,r ( t ) , ξ k,i (cid:105) ≥ m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +0 . 5 , we apply
Revised: Since (cid:104) w ( t, 0) 1 ,r ( t ) , ξ k,i (cid:105) ≥ m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +0 . 5 , we apply

Original: Eq.
Revised: Eq.

Original: (E.7)
Revised: (E.7)

Original: J times to get (cid:104) w ( t,J ) 1 ,r ( t ) , ξ k,i (cid:105) ≥ Θ (cid:0)
Revised: J times to get (cid:104) w ( t,J ) 1 ,r ( t ) , ξ k,i (cid:105) ≥ Θ (cid:0)

Original: m − 1 / 2 (cid:0) 1 + γσ 

Revised: m − 1 / 2 (cid:0) 1 + γσ

Original: ξ d n (cid:1) 0 .
Revised: ξ d n (cid:1) 0 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: Then we have − (cid:96) (cid:48)
Revised: Then we have − (cid:96) (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (E.21)
Revised: (E.21)

Original: The second line is because − (cid:96) (cid:48) ( · ) is non-increasing.
Revised: The second line is because − (cid:96) (cid:48) ( · ) is non-increasing.

Original: To justify the third line, we ﬁrst note that the second part of Theorem E.2 implies (cid:104) w ( t, 0) − 1 ,r (cid:48) , ξ k,i (cid:105)
Revised: To justify the third line, we ﬁrst note that the second part of Theorem E.2 implies (cid:104) w ( t, 0) − 1 ,r (cid:48) , ξ k,i (cid:105)

Original: = (cid:101) O
Revised: = (cid:101) O

Original: ( d − 1 / 2 ) for all r (cid:48) ∈ [ m ] .
Revised: ( d − 1 / 2 ) for all r (cid:48) ∈ [ m ] .

Original: And (cid:104)
Revised: And (cid:104)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2.
Revised: where we used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2.

Original: This implies
Revised: This implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: since J = O (1) and (cid:104) w ( t, 0) − 1 ,r (cid:48) , ξ k,i (cid:105) =
Revised: since J = O (1) and (cid:104) w ( t, 0) − 1 ,r (cid:48) , ξ k,i (cid:105) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: d − 1 / 2 ) .
Revised: d − 1 / 2 ) .

Original: Hence, m
Revised: Hence, m

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (C.1) and Eq.
Revised: (C.1) and Eq.

Original: (E.11) we get that
Revised: (E.11) we get that

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to Lemma D
Revised: where the last equality is due to Lemma D

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: This justiﬁes the third line of Eq.
Revised: This justiﬁes the third line of Eq.

Original: (E.21).
Revised: (E.21).

Original: Using Eq.
Revised: Using Eq.

Original: (E.15), one can directly calculate that T (1) =
Revised: (E.15), one can directly calculate that T (1) =

Original: O (polylog( d )) .
Revised: O (polylog( d )) .

Original: Then, using Eq.
Revised: Then, using Eq.

Original: (C.6) we have
Revised: (C.6) we have

Original: [Equation]
Revised: [Equation]

Original: In the second last line, we can absorb T (1) into (cid:101) O
Revised: In the second last line, we can absorb T (1) into (cid:101) O

Original: ( d − 1 / 2 ) .
Revised: ( d − 1 / 2 ) .

Original: And T (1) ηKn Θ( σ 2 ξ d ) =
Revised: And T (1) ηKn Θ( σ 2 ξ d ) =

Original: (cid:101) Θ(1) .
Revised: (cid:101) Θ(1) .

Original: To get the last line, recall that Ξ ( t (cid:48) )
Revised: To get the last line, recall that Ξ ( t (cid:48) )

Original: k,i ≥ (cid:0) m − 1 / 2 (cid:0) 1 + γσ 

Revised: k,i ≥ (cid:0) m − 1 / 2 (cid:0) 1 + γσ

Original: ξ d n (cid:1) − J +0 .
Revised: ξ d n (cid:1) − J +0 .

Original: 5 (cid:1) =
Revised: 5 (cid:1) =

Original: Ω(1 / polylog( d )) .
Revised: Ω(1 / polylog( d )) .

Original: And (cid:101) Θ(1) O (cid:0) exp (cid:0) − log( d ) 0 .
Revised: And (cid:101) Θ(1) O (cid:0) exp (cid:0) − log( d ) 0 .

Original: 6 (cid:1)(cid:1)(cid:1)
Revised: 6 (cid:1)(cid:1)(cid:1)

Original: =
Revised: =

Original: o (1 / polylog( d )) .
Revised: o (1 / polylog( d )) .

Original: This concludes our proof.
Revised: This concludes our proof.

Original: In short, we have shown that in Phase I, the feature has not been learnt well by any neurons, which is characterized by the inequality Λ ( T (1) )
Revised: In short, we have shown that in Phase I, the feature has not been learnt well by any neurons, which is characterized by the inequality Λ ( T (1) )

Original: j ≤ O ( d − 1 / 2 ) for j ∈ {− 1 , 1 } .
Revised: j ≤ O ( d − 1 / 2 ) for j ∈ {− 1 , 1 } .

Original: And for each noise vector ξ k,i , there exists weights w ( T (1) , 0) j,r with j = y k,i such that their inner product is large: (cid:104) w ( T (1) , 0)
Revised: And for each noise vector ξ k,i , there exists weights w ( T (1) , 0) j,r with j = y k,i such that their inner product is large: (cid:104) w ( T (1) , 0)

Original: j,r , ξ k,i (cid:105) = (cid:101) Θ(1) .
Revised: j,r , ξ k,i (cid:105) = (cid:101) Θ(1) .

Original: E. P HASE II
Revised: E. P HASE II

Original: We have already proven that at the end of Phase I, all of the noises are memorized by the network with an inner product of size (cid:101) Θ(1) whereas the feature is barely memorized, with an inner product of size (cid:101)
Revised: We have already proven that at the end of Phase I, all of the noises are memorized by the network with an inner product of size (cid:101) Θ(1) whereas the feature is barely memorized, with an inner product of size (cid:101)

Original: O ( d − 1 / 2 ) .
Revised: O ( d − 1 / 2 ) .

Original: After Phase I, there is no guarantee that − (cid:96) (cid:48)
Revised: After Phase I, there is no guarantee that − (cid:96) (cid:48)

Original: ( t,J ) k,i = Ω(1) .
Revised: ( t,J ) k,i = Ω(1) .

Original: For this reason, the learning of both the feature and noises will slow down.
Revised: For this reason, the learning of both the feature and noises will slow down.

Original: The difﬁculty in analysis at this stage is that we no longer have − (cid:96) (cid:48) ( t,J ) k,i = Θ(1) .
Revised: The difﬁculty in analysis at this stage is that we no longer have − (cid:96) (cid:48) ( t,J ) k,i = Θ(1) .

Original: The value of (cid:96) (cid:48) ( t,J ) k,i is getting smaller as the inner products
Revised: The value of (cid:96) (cid:48) ( t,J ) k,i is getting smaller as the inner products

Original: (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) gets larger.
Revised: (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) gets larger.

Original: Moreover, if we examine Eq.
Revised: Moreover, if we examine Eq.

Original: (C.6), we observe that the second term on the right hand side could have a non-trivial effect on the growth of the noise inner product, if the third term on the right hand side becomes exponentially smaller than the second term because of (cid:96) (cid:48)
Revised: (C.6), we observe that the second term on the right hand side could have a non-trivial effect on the growth of the noise inner product, if the third term on the right hand side becomes exponentially smaller than the second term because of (cid:96) (cid:48)

Original: ( t,J ) k,i .
Revised: ( t,J ) k,i .

Original: But the sign of each of the summand in the second term is undetermined, which makes the analysis even harder.
Revised: But the sign of each of the summand in the second term is undetermined, which makes the analysis even harder.

Original: We ﬁrst make some claims that will help prove our theorems.
Revised: We ﬁrst make some claims that will help prove our theorems.

Original: And we will verify these claims at the end of this section.
Revised: And we will verify these claims at the end of this section.

Original: Let T = poly( d ) be the total run time.
Revised: Let T = poly( d ) be the total run time.

Original: We have the following claims under Condition D.1 and Condition 4.1.
Revised: We have the following claims under Condition D.1 and Condition 4.1.

Original: Claim E.4.
Revised: Claim E.4.

Original: For any k ∈ [ K ] , i ∈ [ n ] , r ∈ [ m ] , and t ∈ [ T (1) , T ] , let j = − y k,i , then
Revised: For any k ∈ [ K ] , i ∈ [ n ] , r ∈ [ m ] , and t ∈ [ T (1) , T ] , let j = − y k,i , then

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Claim E.5.
Revised: Claim E.5.

Original: For any j ∈ {− 1 , 1 } and t ∈ [ T (1) , T ] we have
Revised: For any j ∈ {− 1 , 1 } and t ∈ [ T (1) , T ] we have

Original: [Equation]
Revised: [Equation]

Original: Claim E.6.
Revised: Claim E.6.

Original: For any j ∈ {− 1 , 1 } , k ∈ [ K ] , r ∈
Revised: For any j ∈ {− 1 , 1 } , k ∈ [ K ] , r ∈

Original: [ m ] and t ∈ [ T (1) , T ] we have
Revised: [ m ] and t ∈ [ T (1) , T ] we have

Original: [Equation]
Revised: [Equation]

Original: Lemma E.7.
Revised: Lemma E.7.

Original: Let t ∈
Revised: Let t ∈

Original: [ T (1) , T ] , k ∈
Revised: [ T (1) , T ] , k ∈

Original: [ K ] , i ∈ [ n ] , j = y k,i and r = arg max r (cid:48) ∈
Revised: [ K ] , i ∈ [ n ] , j = y k,i and r = arg max r (cid:48) ∈

Original: [Equation]
Revised: [Equation]

Original: Suppose Claim E
Revised: Suppose Claim E

Original: [Equation]
Revised: [Equation]

Original: 4 E.5E.6 hold at t .
Revised: 4 E.5E.6 hold at t .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of the ﬁrst inequality is very similar to the ﬁrst part of the proof of Theorem E.3.
Revised: The proof of the ﬁrst inequality is very similar to the ﬁrst part of the proof of Theorem E.3.

Original: Hence we will omit it and focus on proving the second inequality.
Revised: Hence we will omit it and focus on proving the second inequality.

Original: We ﬁrst recall from Eq.
Revised: We ﬁrst recall from Eq.

Original: (C.2)
Revised: (C.2)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: for all τ ∈
Revised: for all τ ∈

Original: [ J − 1] .
Revised: [ J − 1] .

Original: Here we have used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2 to get the inequality.
Revised: Here we have used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2 to get the inequality.

Original: If (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) > m − 1 / 2 , then we have
Revised: If (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) > m − 1 / 2 , then we have

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Ω
Revised: Ω

Original: [Equation]
Revised: [Equation]

Original: The last equality is because m − 1 / 2 (cid:29) J (cid:101) O
Revised: The last equality is because m − 1 / 2 (cid:29) J (cid:101) O

Original: ( d − 1 / 2 ) .
Revised: ( d − 1 / 2 ) .

Original: By the ﬁrst part of this theorem, we have (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≥ Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J (cid:1) .
Revised: By the ﬁrst part of this theorem, we have (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≥ Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J (cid:1) .

Original: Let us consider Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − ( τ +1) (cid:1) ≤ (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≤ O (cid:0)
Revised: Let us consider Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − ( τ +1) (cid:1) ≤ (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≤ O (cid:0)

Original: m − 1 / 2 (cid:0) 1 + γσ 

Revised: m − 1 / 2 (cid:0) 1 + γσ

Original: ξ d n (cid:1) − τ (cid:1) for some τ ∈
Revised: ξ d n (cid:1) − τ (cid:1) for some τ ∈

Original: [ J − 1] .
Revised: [ J − 1] .

Original: Using similar proof as the proof of Hypothesis E.2, it can be shown that − (cid:96) (cid:48)
Revised: Using similar proof as the proof of Hypothesis E.2, it can be shown that − (cid:96) (cid:48)

Original: ( t,τ (cid:48) )
Revised: ( t,τ (cid:48) )

Original: k,i = Ω(1) for all τ (cid:48) ≤ τ if (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≤ Ω (cid:0) m − 1 / 2 (cid:0) 1+ γσ 2 ξ d n (cid:1) − τ (cid:1) .
Revised: k,i = Ω(1) for all τ (cid:48) ≤ τ if (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) ≤ Ω (cid:0) m − 1 / 2 (cid:0) 1+ γσ 2 ξ d n (cid:1) − τ (cid:1) .

Original: Applying Eq.
Revised: Applying Eq.

Original: (E.22)
Revised: (E.22)

Original: τ
Revised: τ

Original: +1 times, we get (cid:104) w ( t,τ +1)
Revised: +1 times, we get (cid:104) w ( t,τ +1)

Original: j,r , ξ k,i (cid:105) ≥ (cid:104) w ( t, 0)
Revised: j,r , ξ k,i (cid:105) ≥ (cid:104) w ( t, 0)

Original: j,r , ξ k,i (cid:105) Ω (cid:0)(cid:0) 

Revised: j,r , ξ k,i (cid:105) Ω (cid:0)(cid:0)

Original: + γσ 2 ξ d n (cid:1)
Revised: + γσ 2 ξ d n (cid:1)

Original: τ +1 (cid:1) , which implies (cid:104) w ( t,τ +1)
Revised: τ +1 (cid:1) , which implies (cid:104) w ( t,τ +1)

Original: j,r
Revised: j,r

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) if
Revised: ) if

Original: [Equation]
Revised: [Equation]

Original: ≥ Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − ( τ +1) (cid:1) .
Revised: ≥ Ω (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − ( τ +1) (cid:1) .

Original: Lastly, Eq.
Revised: Lastly, Eq.

Original: (E.22) also implies
Revised: (E.22) also implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This proves the lemma.
Revised: This proves the lemma.

Original: Lemma E.8.
Revised: Lemma E.8.

Original: Suppose Claim E.4 E.5E.6 hold at t .
Revised: Suppose Claim E.4 E.5E.6 hold at t .

Original: Let k ∈
Revised: Let k ∈

Original: [ K ] , i ∈ [ n ] .
Revised: [ K ] , i ∈ [ n ] .

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then − (cid:96) (cid:48) (
Revised: then − (cid:96) (cid:48) (

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: O ( d − log( d ) 0 . 3 ) .
Revised: O ( d − log( d ) 0 . 3 ) .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, assume y k,i = 1 .
Revised: Without loss of generality, assume y k,i = 1 .

Original: Let r (cid:48) ( t ) =
Revised: Let r (cid:48) ( t ) =

Original: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .
Revised: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Using similar proof as the proof of Hypothesis E.2, it can be shown that − (cid:96) (cid:48)
Revised: Using similar proof as the proof of Hypothesis E.2, it can be shown that − (cid:96) (cid:48)

Original: ( t,τ ) k,i = Ω(1) for all τ ∈
Revised: ( t,τ ) k,i = Ω(1) for all τ ∈

Original: [ J − 1] under Claim E.5E.6.
Revised: [ J − 1] under Claim E.5E.6.

Original: We then apply Eq.
Revised: We then apply Eq.

Original: (E.22)
Revised: (E.22)

Original: J − 1 times to get
Revised: J − 1 times to get

Original: [Equation]
Revised: [Equation]

Original: The third line is because by assumption we have Ξ
Revised: The third line is because by assumption we have Ξ

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: J
Revised: J

Original: [Equation]
Revised: [Equation]

Original: O ( d − 1 / 2 ) .
Revised: O ( d − 1 / 2 ) .

Original: On the other hand, Claim E.4, tells us that
Revised: On the other hand, Claim E.4, tells us that

Original: [Equation]
Revised: [Equation]

Original: for all r ∈ [ m ] .
Revised: for all r ∈ [ m ] .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (C.1) and Eq.
Revised: (C.1) and Eq.

Original: (E.11) we get that
Revised: (E.11) we get that

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: We now have
Revised: We now have

Original: [Equation]
Revised: [Equation]

Original: This proves the lemma
Revised: This proves the lemma

Original: [Equation]
Revised: [Equation]

Original: Theorem E.9.
Revised: Theorem E.9.

Original: If Claim E.4E.5E.6 hold for all T (1) ≤ t ≤ T − 1 , then
Revised: If Claim E.4E.5E.6 hold for all T (1) ≤ t ≤ T − 1 , then

Original: [Equation]
Revised: [Equation]

Original: for all k ∈ [ K ] , i ∈ [ n ] .
Revised: for all k ∈ [ K ] , i ∈ [ n ] .

Original: Proof.
Revised: Proof.

Original: Set A ( q ) = m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − q for q ∈ [ J ] .
Revised: Set A ( q ) = m − 1 / 2 (cid:16) 1 + γσ 2 ξ d n (cid:17) − q for q ∈ [ J ] .

Original: We proved in Theorem E.3 that Ω( A ( J )) ≤ Ξ ( T (1) ) k,i ≤ O ( A ( J − 0 . 5)) for all k ∈
Revised: We proved in Theorem E.3 that Ω( A ( J )) ≤ Ξ ( T (1) ) k,i ≤ O ( A ( J − 0 . 5)) for all k ∈

Original: [ K ] , i ∈ [ n ] .
Revised: [ K ] , i ∈ [ n ] .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: where the inﬁmum over an empty set is deﬁned as ∞ .
Revised: where the inﬁmum over an empty set is deﬁned as ∞ .

Original: In words, t ( k, i ) is the last iteration with Ξ ( t ) k,i ≤
Revised: In words, t ( k, i ) is the last iteration with Ξ ( t ) k,i ≤

Original: A ( J − 1) .
Revised: A ( J − 1) .

Original: We have shown in the proof of Theorem E.3 that Ξ ( t ( k,i )+1) k,i < 1 .
Revised: We have shown in the proof of Theorem E.3 that Ξ ( t ( k,i )+1) k,i < 1 .

Original: 5 A ( J − 1) .
Revised: 5 A ( J − 1) .

Original: Similarly deﬁne
Revised: Similarly deﬁne

Original: [Equation]
Revised: [Equation]

Original: Again the inf over an empty set is deﬁned as ∞ .
Revised: Again the inf over an empty set is deﬁned as ∞ .

Original: So ˜ t ( k, i ) is the last iteration with Ξ ( t ) k,i ≤ 4 A ( J − 1) .
Revised: So ˜ t ( k, i ) is the last iteration with Ξ ( t ) k,i ≤ 4 A ( J − 1) .

Original: Our goal is to show that ˜ t ( k, i )
Revised: Our goal is to show that ˜ t ( k, i )

Original: = ∞ for any k ∈ [ K ] and i ∈
Revised: = ∞ for any k ∈ [ K ] and i ∈

Original: [ n ] .
Revised: [ n ] .

Original: We will prove this by contradiction.
Revised: We will prove this by contradiction.

Original: Let us suppose that ˜ t ( k, i ) < ∞ .
Revised: Let us suppose that ˜ t ( k, i ) < ∞ .

Original: Then, for any t ∈ [ t ( k, i ) + 1 , ˜ t ( k, i )] we have that − (cid:96) (cid:48)
Revised: Then, for any t ∈ [ t ( k, i ) + 1 , ˜ t ( k, i )] we have that − (cid:96) (cid:48)

Original: ( t,J ) k,
Revised: ( t,J ) k,

Original: i =
Revised: i =

Original: O ( d − log( d ) 0 . 3 ) by Lemma E.8.
Revised: O ( d − log( d ) 0 . 3 ) by Lemma E.8.

Original: Recall
Revised: Recall

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because σ (cid:48) ( · ) ≤
Revised: where the second inequality is because σ (cid:48) ( · ) ≤

Original: [Equation]
Revised: [Equation]

Original: And similarly we have
Revised: And similarly we have

Original: [Equation]
Revised: [Equation]

Original: where we have again used σ (cid:48)
Revised: where we have again used σ (cid:48)

Original: ( · ) ≤ 1 to the second inequality.
Revised: ( · ) ≤ 1 to the second inequality.

Original: We have the following bound due to
Revised: We have the following bound due to

Original: Lemma E.
Revised: Lemma E.

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: where the last equality is because T = O (poly( d )) .
Revised: where the last equality is because T = O (poly( d )) .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Suppose it holds that
Revised: Suppose it holds that

Original: [Equation]
Revised: [Equation]

Original: Then using Eq.
Revised: Then using Eq.

Original: (E.24) we get
Revised: (E.24) we get

Original: [Equation]
Revised: [Equation]

Original: The second inequality is due to Eq.
Revised: The second inequality is due to Eq.

Original: (E.25), Eq.
Revised: (E.25), Eq.

Original: (E.26) and the deﬁnition of ( k (cid:48) , i (cid:48) ) .
Revised: (E.26) and the deﬁnition of ( k (cid:48) , i (cid:48) ) .

Original: The third inequality is because η Θ( σ 2 ξ √ d (cid:112) log( Kn ))
Revised: The third inequality is because η Θ( σ 2 ξ √ d (cid:112) log( Kn ))

Original: d 1 / 4 + o ( d − 1 ) =
Revised: d 1 / 4 + o ( d − 1 ) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: ( d − 1 / 4 ) and Ξ ( t ( k,i )+1)
Revised: ( d − 1 / 4 ) and Ξ ( t ( k,i )+1)

Original: k,i ≥ A ( J − 1)
Revised: k,i ≥ A ( J − 1)

Original: =
Revised: =

Original: (cid:101) Θ(1) .
Revised: (cid:101) Θ(1) .

Original: Thus Eq.
Revised: Thus Eq.

Original: (E.26) contradicts Ξ (˜ t ( k,i )
Revised: (E.26) contradicts Ξ (˜ t ( k,i )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ≥ 4 A ( J − 1) and we must have ˜ t ( k, i ) = ∞ .
Revised: ≥ 4 A ( J − 1) and we must have ˜ t ( k, i ) = ∞ .

Original: Next, we will show that if Eq.
Revised: Next, we will show that if Eq.

Original: (E.26) does not hold, then we also have ˜ t ( k, i )
Revised: (E.26) does not hold, then we also have ˜ t ( k, i )

Original: = ∞ .
Revised: = ∞ .

Original: Let us suppose
Revised: Let us suppose

Original: [Equation]
Revised: [Equation]

Original: Since − (cid:96) (cid:48) ( t,J ) k (cid:48) ,i (cid:48) > 0 for any t , we get that (cid:80) t 0 t
Revised: Since − (cid:96) (cid:48) ( t,J ) k (cid:48) ,i (cid:48) > 0 for any t , we get that (cid:80) t 0 t

Original: = t ( k,i )+1 − (cid:96) (cid:48)
Revised: = t ( k,i )+1 − (cid:96) (cid:48)

Original: ( t,J ) k (cid:48) ,i (cid:48) is increasing in t 0 .
Revised: ( t,J ) k (cid:48) ,i (cid:48) is increasing in t 0 .

Original: Let t ( k, i ) + 1 < t 0 <
Revised: Let t ( k, i ) + 1 < t 0 <

Original: ˜ t ( k, i ) be the ﬁrst iteration such that
Revised: ˜ t ( k, i ) be the ﬁrst iteration such that

Original: [Equation]
Revised: [Equation]

Original: Since each − (cid:96) (cid:48) ( t,J ) k (cid:48) ,i (cid:48) < 1 , we get that t 0 (cid:88)
Revised: Since each − (cid:96) (cid:48) ( t,J ) k (cid:48) ,i (cid:48) < 1 , we get that t 0 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ,i
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: E.27) where the second and third line are by deﬁnition of t 0 .
Revised: E.27) where the second and third line are by deﬁnition of t 0 .

Original: We can sum Eq.
Revised: We can sum Eq.

Original: (E.23) over time to obtain t 0 (cid:88)
Revised: (E.23) over time to obtain t 0 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where the second inequality is because t 0 < ˜ t ( k, i ) .
Revised: where the second inequality is because t 0 < ˜ t ( k, i ) .

Original: We have used the deﬁnition of ( k (cid:48) , i (cid:48) ) to get the last inequality.
Revised: We have used the deﬁnition of ( k (cid:48) , i (cid:48) ) to get the last inequality.

Original: Then, subtracting (cid:80) t 0 t = t ( k,i )
Revised: Then, subtracting (cid:80) t 0 t = t ( k,i )

Original: +1 Ξ ( t ) k (cid:48) ,i (cid:48) from both sides yields
Revised: +1 Ξ ( t ) k (cid:48) ,i (cid:48) from both sides yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we used Lemma E.7 to get the ﬁrst inequality.
Revised: where we used Lemma E.7 to get the ﬁrst inequality.

Original: The ﬁrst equality is due to Eq. (E.27).
Revised: The ﬁrst equality is due to Eq. (E.27).

Original: The last inequality is by our assumption (cid:80) ˜ t ( k,i ) t
Revised: The last inequality is by our assumption (cid:80) ˜ t ( k,i ) t

Original: = t ( k,i )+1 − (cid:96) (cid:48)
Revised: = t ( k,i )+1 − (cid:96) (cid:48)

Original: ( t,J ) k (cid:48) ,i (cid:48) ≥
Revised: ( t,J ) k (cid:48) ,i (cid:48) ≥

Original: d 1 / 4 .
Revised: d 1 / 4 .

Original: It is then straightforward to check that
Revised: It is then straightforward to check that

Original: [Equation]
Revised: [Equation]

Original: ) , for all t 0 + 1 ≤ t ≤ ˜ t ( k, i ) .
Revised: ) , for all t 0 + 1 ≤ t ≤ ˜ t ( k, i ) .

Original: To see this, let us sum Eq.
Revised: To see this, let us sum Eq.

Original: (E.23) over time to obtain
Revised: (E.23) over time to obtain

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Kn
Revised: Kn

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because we are removing a positive term from the sum.
Revised: where the second inequality is because we are removing a positive term from the sum.

Original: The third inequality is because t − 1 < ˜ t ( k, i ) and t 0 > t ( k, i ) .
Revised: The third inequality is because t − 1 < ˜ t ( k, i ) and t 0 > t ( k, i ) .

Original: The fourth inequality is by deﬁnition of ( k (cid:48) , i (cid:48) ) .
Revised: The fourth inequality is by deﬁnition of ( k (cid:48) , i (cid:48) ) .

Original: The second last inequality is by the second last line of Eq.
Revised: The second last inequality is by the second last line of Eq.

Original: (E.28).
Revised: (E.28).

Original: And our last inequality follows
Revised: And our last inequality follows

Original: Eq.
Revised: Eq.

Original: (E.28).
Revised: (E.28).

Original: Then, using similar proof of Lemma E.8, we have − (cid:96) (cid:48)
Revised: Then, using similar proof of Lemma E.8, we have − (cid:96) (cid:48)

Original: ( t,J ) k
Revised: ( t,J ) k

Original: [Equation]
Revised: [Equation]

Original: ,i
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: = O (exp( − (cid:101)
Revised: = O (exp( − (cid:101)

Original: Θ( d 1 / 8 )))
Revised: Θ( d 1 / 8 )))

Original: for all t 0 + 1 ≤ t ≤ ˜ t ( k, i ) .
Revised: for all t 0 + 1 ≤ t ≤ ˜ t ( k, i ) .

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: where we used Eq.
Revised: where we used Eq.

Original: (E.27) to get the second line.
Revised: (E.27) to get the second line.

Original: And we used T
Revised: And we used T

Original: = O (poly( d )) to get the fourth line.
Revised: = O (poly( d )) to get the fourth line.

Original: To obtain the last inequality, we recall the assumption (cid:80) ˜ t ( k,i ) t
Revised: To obtain the last inequality, we recall the assumption (cid:80) ˜ t ( k,i ) t

Original: = t ( k,i )+1 − (cid:96) (cid:48)
Revised: = t ( k,i )+1 − (cid:96) (cid:48)

Original: ( t,J ) k (cid:48) ,i (cid:48) >
Revised: ( t,J ) k (cid:48) ,i (cid:48) >

Original: d 1 / 4 , which implies o ( d − 1 ) <
Revised: d 1 / 4 , which implies o ( d − 1 ) <

Original: o ( d − 1 ) (cid:80) ˜ t ( k,i ) t
Revised: o ( d − 1 ) (cid:80) ˜ t ( k,i ) t

Original: = t ( k,i )+1 − (cid:96) (cid:48)
Revised: = t ( k,i )+1 − (cid:96) (cid:48)

Original: ( t,J ) k (cid:48) ,i (cid:48) .
Revised: ( t,J ) k (cid:48) ,i (cid:48) .

Original: Eq.
Revised: Eq.

Original: (E.29) is a contradiction in the form of x ≤ Ax for some x ≥ 0 and A < 1 .
Revised: (E.29) is a contradiction in the form of x ≤ Ax for some x ≥ 0 and A < 1 .

Original: This implies that under the assumption (cid:80) ˜ t ( k,i ) t
Revised: This implies that under the assumption (cid:80) ˜ t ( k,i ) t

Original: = t ( k,i )+1 − (cid:96) (cid:48)
Revised: = t ( k,i )+1 − (cid:96) (cid:48)

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: we should still have ˜ t ( k, i )
Revised: we should still have ˜ t ( k, i )

Original: = ∞ .
Revised: = ∞ .

Original: Combining with our previous result which says ˜ t ( k, i )
Revised: Combining with our previous result which says ˜ t ( k, i )

Original: = ∞ when
Revised: = ∞ when

Original: [Equation]
Revised: [Equation]

Original: , we conclude that ˜ t ( k, i )
Revised: , we conclude that ˜ t ( k, i )

Original: = ∞
Revised: = ∞

Original: and we are done.
Revised: and we are done.

Original: Now that we know the noise inner products are upper bounded by O (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +1 (cid:1) , we can provide a convergence guarantee based on this upper bound.
Revised: Now that we know the noise inner products are upper bounded by O (cid:0) m − 1 / 2 (cid:0) 1 + γσ 2 ξ d n (cid:1) − J +1 (cid:1) , we can provide a convergence guarantee based on this upper bound.

Original: Theorem E.10 (Restatement of Theorem 4.4 (1)) .
Revised: Theorem E.10 (Restatement of Theorem 4.4 (1)) .

Original: Under Condition D.1 and Condition 4.1, let
Revised: Under Condition D.1 and Condition 4.1, let

Original: T = poly( d ) be the total number of iterations.
Revised: T = poly( d ) be the total number of iterations.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Theorem E.9, we know that
Revised: By Theorem E.9, we know that

Original: [Equation]
Revised: [Equation]

Original: for all k ∈ [ K ] and i ∈ [ n ] .
Revised: for all k ∈ [ K ] and i ∈ [ n ] .

Original: Denote Ξ ( t ) =
Revised: Denote Ξ ( t ) =

Original: (cid:80) k,
Revised: (cid:80) k,

Original: i Ξ ( t ) k,i .
Revised: i Ξ ( t ) k,i .

Original: It then follows that Ξ ( T ) ≤
Revised: It then follows that Ξ ( T ) ≤

Original: Kn (cid:101) O (1) =
Revised: Kn (cid:101) O (1) =

Original: (cid:101) O (1) .
Revised: (cid:101) O (1) .

Original: Moreover, if we sum Eq.
Revised: Moreover, if we sum Eq.

Original: (E.24) over all k, i , we get Ξ ( t +1) ≥ Ξ ( t )
Revised: (E.24) over all k, i , we get Ξ ( t +1) ≥ Ξ ( t )

Original: + η ( Kn − 1) (cid:88)
Revised: + η ( Kn − 1) (cid:88)

Original: ( k,i
Revised: ( k,i

Original: [Equation]
Revised: [Equation]

Original: (cid:96) (cid:48)
Revised: (cid:96) (cid:48)

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: J ) k,i Θ( σ 2 ξ √ d (cid:112) log( Kn ))
Revised: J ) k,i Θ( σ 2 ξ √ d (cid:112) log( Kn ))

Original: − η Kn (cid:88)
Revised: − η Kn (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the Ω (cid:0) 1 √ m (cid:1) term is by Lemma E.7.
Revised: where the Ω (cid:0) 1 √ m (cid:1) term is by Lemma E.7.

Original: We can then sum over t to obtain
Revised: We can then sum over t to obtain

Original: [Equation]
Revised: [Equation]

Original: The last inequality is because Ξ ( T ) − Ξ ( T
Revised: The last inequality is because Ξ ( T ) − Ξ ( T

Original: [Equation]
Revised: [Equation]

Original: ) ≤ (cid:101) O (1) .
Revised: ) ≤ (cid:101) O (1) .

Original: And we can bound the minimum over t by
Revised: And we can bound the minimum over t by

Original: [Equation]
Revised: [Equation]

Original: where the second equality is because T (1) =
Revised: where the second equality is because T (1) =

Original: polylog( d ) =
Revised: polylog( d ) =

Original: o ( T ) .
Revised: o ( T ) .

Original: Denote −
Revised: Denote −

Original: [Equation]
Revised: [Equation]

Original: k,i = − (cid:96) (cid:48)
Revised: k,i = − (cid:96) (cid:48)

Original: ( α ( t, J, k, i )) , i.e. the argument of − (cid:96) (cid:48) ( t,J ) k,i is α ( t, J, k, i ) .
Revised: ( α ( t, J, k, i )) , i.e. the argument of − (cid:96) (cid:48) ( t,J ) k,i is α ( t, J, k, i ) .

Original: Then the loss is related to − (cid:96) (cid:48)
Revised: Then the loss is related to − (cid:96) (cid:48)

Original: ( t,J ) k,i by (cid:98) L tr-tr ( W ( t ) )
Revised: ( t,J ) k,i by (cid:98) L tr-tr ( W ( t ) )

Original: = (cid:88)
Revised: = (cid:88)

Original: k,i (cid:96) (cid:0) α ( t, J, k, i ) (cid:1) .
Revised: k,i (cid:96) (cid:0) α ( t, J, k, i ) (cid:1) .

Original: Let t (cid:48) =
Revised: Let t (cid:48) =

Original: arg min t (cid:80) k,i − (cid:96) (cid:48) ( t,J ) k,i .
Revised: arg min t (cid:80) k,i − (cid:96) (cid:48) ( t,J ) k,i .

Original: By deﬁnition of (cid:96) (cid:48) ( · ) and (cid:96) ( · ) we can bound − (cid:96) (cid:48)
Revised: By deﬁnition of (cid:96) (cid:48) ( · ) and (cid:96) ( · ) we can bound − (cid:96) (cid:48)

Original: ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 .
Revised: ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 .

Original: This can be used to bound our loss by
Revised: This can be used to bound our loss by

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: This proves our theorem.
Revised: This proves our theorem.

Original: It remains to prove Claims E.4E.5E.6.
Revised: It remains to prove Claims E.4E.5E.6.

Original: Proof of Claim E.4.
Revised: Proof of Claim E.4.

Original: Without loss of generality, assume y k,i = 1 .
Revised: Without loss of generality, assume y k,i = 1 .

Original: By Eq.
Revised: By Eq.

Original: (E.16), we have the following upper bound (cid:104)
Revised: (E.16), we have the following upper bound (cid:104)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where the second inequality is because we are adding a non-negative term to the sum.
Revised: where the second inequality is because we are adding a non-negative term to the sum.

Original: The last inequality is because σ (cid:48) ( · ) ≤ 1 .
Revised: The last inequality is because σ (cid:48) ( · ) ≤ 1 .

Original: Suppose all of our claims hold for t ≤ T (cid:48) , where T (cid:48)
Revised: Suppose all of our claims hold for t ≤ T (cid:48) , where T (cid:48)

Original: ≤ T − 1 .
Revised: ≤ T − 1 .

Original: Using a similar derivation of Eq.
Revised: Using a similar derivation of Eq.

Original: (E.30) we have
Revised: (E.30) we have

Original: [Equation]
Revised: [Equation]

Original: Then we get
Revised: Then we get

Original: [Equation]
Revised: [Equation]

Original: where the third line is due to part (2) of Theorem E.2.
Revised: where the third line is due to part (2) of Theorem E.2.

Original: This proves our claim.
Revised: This proves our claim.

Original: Proof of Claim E.5.
Revised: Proof of Claim E.5.

Original: We will show that for any T (cid:48) ≤ T =
Revised: We will show that for any T (cid:48) ≤ T =

Original: O (poly( d )) , we have that Λ ( T (cid:48) )
Revised: O (poly( d )) , we have that Λ ( T (cid:48) )

Original: j =
Revised: j =

Original: O ( d − 1 / 4 ) for j ∈ {− 1 , 1 } .
Revised: O ( d − 1 / 4 ) for j ∈ {− 1 , 1 } .

Original: We have the following upper bound
Revised: We have the following upper bound

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Suppose that there exists some ˜ t ≤ T (cid:48) such that Λ (˜ t )
Revised: Suppose that there exists some ˜ t ≤ T (cid:48) such that Λ (˜ t )

Original: j ≥ d − 1 / 4 for the ﬁrst time.
Revised: j ≥ d − 1 / 4 for the ﬁrst time.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The third inequality is because by the ﬁrst part of Theorem E.2, we have Λ ( T (1) )
Revised: The third inequality is because by the ﬁrst part of Theorem E.2, we have Λ ( T (1) )

Original: j = (cid:101)
Revised: j = (cid:101)

Original: O ( d − 1 / 2 ) .
Revised: O ( d − 1 / 2 ) .

Original: And hence d − 1 / 4 / Λ ( T (1) )
Revised: And hence d − 1 / 4 / Λ ( T (1) )

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: Ω
Revised: Ω

Original: [Equation]
Revised: [Equation]

Original: d 1 / 4 ) .
Revised: d 1 / 4 ) .

Original: Note that in the proof of Theorem E.10, we have that
Revised: Note that in the proof of Theorem E.10, we have that

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , which creates a contradiction.
Revised: (cid:17) , which creates a contradiction.

Original: Hence, for all ˜ t ≤ T (cid:48) , we have Λ (˜ t )
Revised: Hence, for all ˜ t ≤ T (cid:48) , we have Λ (˜ t )

Original: j ≤ d − 1 / 4 .
Revised: j ≤ d − 1 / 4 .

Original: This proves the claim.
Revised: This proves the claim.

Original: Proof of Claim E.6.
Revised: Proof of Claim E.6.

Original: By our result in Phase I, we have that Claim E.6 holds at T (1) .
Revised: By our result in Phase I, we have that Claim E.6 holds at T (1) .

Original: The rest of the proof is similar to the proof of Hypothesis E.1 and we will omit it here.
Revised: The rest of the proof is similar to the proof of Hypothesis E.1 and we will omit it here.

Original: We have shown that for T = poly( d ) , the feature has not been learnt well if one uses the train-train method.
Revised: We have shown that for T = poly( d ) , the feature has not been learnt well if one uses the train-train method.

Original: Thus the testing loss will be large, which is characterized by the following theorem.
Revised: Thus the testing loss will be large, which is characterized by the following theorem.

Original: Theorem E.11 (Restatement of Theorem 4.4 (2)) .
Revised: Theorem E.11 (Restatement of Theorem 4.4 (2)) .

Original: Under Condition 4.1 and Condition D.1, the test loss is large throughout the whole training process
Revised: Under Condition 4.1 and Condition D.1, the test loss is large throughout the whole training process

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Sample a new example x from the distribution, since we use the convolutional structure, we can assume that the ﬁrst patch x (1) =
Revised: Sample a new example x from the distribution, since we use the convolutional structure, we can assume that the ﬁrst patch x (1) =

Original: y · ( ν + z ) and the second patch x (2) = ξ .
Revised: y · ( ν + z ) and the second patch x (2) = ξ .

Original: Clearly (cid:104) ξ k,i , ξ (cid:105) follows the Gaussian distribution with mean zero and standard deviation σ ξ · (cid:107) ξ k,i (cid:107) 2 .
Revised: Clearly (cid:104) ξ k,i , ξ (cid:105) follows the Gaussian distribution with mean zero and standard deviation σ ξ · (cid:107) ξ k,i (cid:107) 2 .

Original: By
Revised: By

Original: Lemma D.2, we further know that (cid:107) ξ k,i (cid:107) 2 ≤ 2 σ ξ √ d .
Revised: Lemma D.2, we further know that (cid:107) ξ k,i (cid:107) 2 ≤ 2 σ ξ √ d .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Similarly we have that
Revised: Similarly we have that

Original: [Equation]
Revised: [Equation]

Original: Denote E to be the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w (0 , 0)
Revised: Denote E to be the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w (0 , 0)

Original: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈
Revised: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈

Original: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } .
Revised: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } .

Original: Applying union bound, we have that the P ( E ) ≥ 1 − 2( nK + m ) exp( − d 1 / 4 ) ≥ 1 / 2 .
Revised: Applying union bound, we have that the P ( E ) ≥ 1 − 2( nK + m ) exp( − d 1 / 4 ) ≥ 1 / 2 .

Original: We can divide L test ( W ) into two parts:
Revised: We can divide L test ( W ) into two parts:

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Let T (cid:48) ≤ T .
Revised: Let T (cid:48) ≤ T .

Original: And suppose that T (cid:48) ≥ T (1) .
Revised: And suppose that T (cid:48) ≥ T (1) .

Original: When event E holds, for any j ∈ {− 1 , 1 } , we have (cid:104) w ( T (cid:48) , 0) j,r , ξ (cid:105) =
Revised: When event E holds, for any j ∈ {− 1 , 1 } , we have (cid:104) w ( T (cid:48) , 0) j,r , ξ (cid:105) =

Original: [Equation]
Revised: [Equation]

Original: where the second line is by Eq.
Revised: where the second line is by Eq.

Original: (E.31).
Revised: (E.31).

Original: The third line is by 0 ≤ σ (cid:48) ( · ) ≤ 1 .
Revised: The third line is by 0 ≤ σ (cid:48) ( · ) ≤ 1 .

Original: The third last line is due to − (cid:96) (cid:48) ( · ) ≤ 1 , T (2) =
Revised: The third last line is due to − (cid:96) (cid:48) ( · ) ≤ 1 , T (2) =

Original: polylog( d ) and (cid:80) T − 1 t
Revised: polylog( d ) and (cid:80) T − 1 t

Original: =
Revised: =

Original: T (2) (cid:80) ( k,i )
Revised: T (2) (cid:80) ( k,i )

Original: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i = (cid:101) O (1) .
Revised: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i = (cid:101) O (1) .

Original: The second last line is by Eq.
Revised: The second last line is by Eq.

Original: (E.32).
Revised: (E.32).

Original: We note that Eq.
Revised: We note that Eq.

Original: (E.34) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .
Revised: (E.34) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .

Original: Using the same procedure, one can show that j (cid:104) w ( T (cid:48) , 0) j,r , z (cid:105) ≤ (cid:101) O
Revised: Using the same procedure, one can show that j (cid:104) w ( T (cid:48) , 0) j,r , z (cid:105) ≤ (cid:101) O

Original: ( d − 1 / 4 ) for all r ∈ [ m ] and j ∈ {− 1 , 1 } .
Revised: ( d − 1 / 4 ) for all r ∈ [ m ] and j ∈ {− 1 , 1 } .

Original: Now without loss of generality assume y = 1 .
Revised: Now without loss of generality assume y = 1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: ≥ (cid:96) (cid:16) F +

Revised: ≥ (cid:96) (cid:16) F +

Original: ( W ( T (cid:48) )
Revised: ( W ( T (cid:48) )

Original: +1 , x ) (cid:17) =
Revised: +1 , x ) (cid:17) =

Original: (cid:96) (cid:18) m (cid:88) r
Revised: (cid:96) (cid:18) m (cid:88) r

Original: = σ ( (cid:104) w ( T (cid:48) , 0) 1 ,r , ν
Revised: = σ ( (cid:104) w ( T (cid:48) , 0) 1 ,r , ν

Original: + z (cid:105) )
Revised: + z (cid:105) )

Original: + σ ( (cid:104) w ( T (cid:48) , 0) 1 ,r , ξ (cid:105) )
Revised: + σ ( (cid:104) w ( T (cid:48) , 0) 1 ,r , ξ (cid:105) )

Original: (cid:19) ≥ (cid:96) (cid:18) m (cid:88) r
Revised: (cid:19) ≥ (cid:96) (cid:18) m (cid:88) r

Original: = σ (cid:0) (cid:101) O
Revised: = σ (cid:0) (cid:101) O

Original: ( d − 1 / 4 ) + (cid:101)
Revised: ( d − 1 / 4 ) + (cid:101)

Original: O
Revised: O

Original: ( d − 1 / 4 ) (cid:1) + σ (cid:0) (cid:101)
Revised: ( d − 1 / 4 ) (cid:1) + σ (cid:0) (cid:101)

Original: O ( d − 1 / 4 ) (cid:1)(cid:19) =
Revised: O ( d − 1 / 4 ) (cid:1)(cid:19) =

Original: (cid:96) (cid:16) m (cid:101) O
Revised: (cid:96) (cid:16) m (cid:101) O

Original: ( d − 1 / 4 ) (cid:17) = Ω(1) .
Revised: ( d − 1 / 4 ) (cid:17) = Ω(1) .

Original: (E.35) where second line is because (cid:96) ( · ) is nonincreasing and the fourth line is due to Claim E.5.
Revised: (E.35) where second line is because (cid:96) ( · ) is nonincreasing and the fourth line is due to Claim E.5.

Original: Finally plugging Eq.
Revised: Finally plugging Eq.

Original: (E.35) into Eq.
Revised: (E.35) into Eq.

Original: (E.33) gives that
Revised: (E.33) gives that

Original: [Equation]
Revised: [Equation]

Original: Since T (cid:48) ∈
Revised: Since T (cid:48) ∈

Original: [ T (1) , T ] is arbitrary, we get that
Revised: [ T (1) , T ] is arbitrary, we get that

Original: [Equation]
Revised: [Equation]

Original: However, the above proof would also work for T (cid:48) <
Revised: However, the above proof would also work for T (cid:48) <

Original: T (1) using our proofs from Phase I. Hence,
Revised: T (1) using our proofs from Phase I. Hence,

Original: [Equation]
Revised: [Equation]

Original: which concludes our theorem.
Revised: which concludes our theorem.

Original: [Equation]
Revised: [Equation]

Original: Contrary to the train-train method, we will show in this section that under Condition D.1, the feature will be learned by our neural network.
Revised: Contrary to the train-train method, we will show in this section that under Condition D.1, the feature will be learned by our neural network.

Original: Abusing notations, let Ξ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) :
Revised: Abusing notations, let Ξ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) :

Original: r ∈ [ m ] (cid:111) , Γ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) − 1 ,r , ξ k,i (cid:105) : r ∈ [ m ] (cid:111) , and Λ ( t ) j = max (cid:110)
Revised: r ∈ [ m ] (cid:111) , Γ ( t ) k,i = max (cid:110) (cid:104) w ( t, 0) − 1 ,r , ξ k,i (cid:105) : r ∈ [ m ] (cid:111) , and Λ ( t ) j = max (cid:110)

Original: j (cid:104) w ( t, 0) j,r , ν (cid:105) :
Revised: j (cid:104) w ( t, 0) j,r , ν (cid:105) :

Original: r ∈ [ m ] (cid:111) for j ∈ {− 1 , 1 } .
Revised: r ∈ [ m ] (cid:111) for j ∈ {− 1 , 1 } .

Original: Deﬁne C =
Revised: Deﬁne C =

Original: Jγσ 2 ξ √ d (cid:112) log( Kn ) , (cid:98) Ξ ( t ) k,i = max { C, Ξ ( t ) k,i } , and (cid:98) Γ ( t )
Revised: Jγσ 2 ξ √ d (cid:112) log( Kn ) , (cid:98) Ξ ( t ) k,i = max { C, Ξ ( t ) k,i } , and (cid:98) Γ ( t )

Original: k,i = max { C, Γ ( t ) k,i } .
Revised: k,i = max { C, Γ ( t ) k,i } .

Original: Let Ψ
Revised: Let Ψ

Original: =
Revised: =

Original: ( k, i ) :
Revised: ( k, i ) :

Original: k ∈ [ K ] , i > n 1 } , Ψ + = { ( k, i ) : k ∈ [ K ] , i > n 1 , y k,i = 1 } , and Ψ − = { ( k, i ) :
Revised: k ∈ [ K ] , i > n 1 } , Ψ + = { ( k, i ) : k ∈ [ K ] , i > n 1 , y k,i = 1 } , and Ψ − = { ( k, i ) :

Original: k ∈ [ K ] , i > n 1 , y k,i = − 1 } .
Revised: k ∈ [ K ] , i > n 1 , y k,i = − 1 } .

Original: So Ψ represents all of the samples from the validation set of all tasks.
Revised: So Ψ represents all of the samples from the validation set of all tasks.

Original: And Ψ j represents the samples from the validation set of all tasks with label j .
Revised: And Ψ j represents the samples from the validation set of all tasks with label j .

Original: We ﬁrst present a series of useful lemmas that hold for all t ≥ 0 .
Revised: We ﬁrst present a series of useful lemmas that hold for all t ≥ 0 .

Original: Theses lemmas basically show that the learning speed of z k are very slow compared with the learning speed of the feature ν .
Revised: Theses lemmas basically show that the learning speed of z k are very slow compared with the learning speed of the feature ν .

Original: Lemma F.1.
Revised: Lemma F.1.

Original: Under Condition D.1 and Condition 4.1, for any t ≥ 0 , k ∈ [ M ] , j ∈ {− 1 , 1 } , if one uses the train-validation method, then we have
Revised: Under Condition D.1 and Condition 4.1, for any t ≥ 0 , k ∈ [ M ] , j ∈ {− 1 , 1 } , if one uses the train-validation method, then we have

Original: [Equation]
Revised: [Equation]

Original: j (cid:104) w ( t, 0) j,r , z k (cid:105) (cid:12)(cid:12)(cid:12) =
Revised: j (cid:104) w ( t, 0) j,r , z k (cid:105) (cid:12)(cid:12)(cid:12) =

Original: o (1) Λ ( t ) j .
Revised: o (1) Λ ( t ) j .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let j = 1 , k ∈
Revised: Without loss of generality, let j = 1 , k ∈

Original: [ M ] .
Revised: [ M ] .

Original: Denote r ( t ) =
Revised: Denote r ( t ) =

Original: arg max r ∈ [ m ] (cid:12)(cid:12) (cid:104) w ( t, 0) 1 ,r , z k (cid:105) (cid:12)(cid:12) .
Revised: arg max r ∈ [ m ] (cid:12)(cid:12) (cid:104) w ( t, 0) 1 ,r , z k (cid:105) (cid:12)(cid:12) .

Original: Deﬁne
Revised: Deﬁne

Original: D ( t ) :
Revised: D ( t ) :

Original: = (cid:12)(cid:12) (cid:104) w ( t, 0) 1 ,r ( t ) , z k (cid:105) (cid:12)(cid:12) Λ ( t ).
Revised: = (cid:12)(cid:12) (cid:104) w ( t, 0) 1 ,r ( t ) , z k (cid:105) (cid:12)(cid:12) Λ ( t ).

Original: At t = 0 , by Lemma D.2 we have
Revised: At t = 0 , by Lemma D.2 we have

Original: D (0) =
Revised: D (0) =

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , z
Revised: , z

Original: [Equation]
Revised: [Equation]

Original: (cid:105) (cid:12)(cid:12) Λ (0)
Revised: (cid:105) (cid:12)(cid:12) Λ (0)

Original: ≤ Θ( σ s σ√ d (cid:112) log( mKn ))
Revised: ≤ Θ( σ s σ√ d (cid:112) log( mKn ))

Original: Θ( σ 0 ) =
Revised: Θ( σ 0 ) =

Original: Θ( σ s √ d (cid:112) log( mKn ))
Revised: Θ( σ s √ d (cid:112) log( mKn ))

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: Deﬁne D 0 :
Revised: Deﬁne D 0 :

Original: =
Revised: =

Original: Θ( σ s √ d (cid:112) log( mKn )) .
Revised: Θ( σ s √ d (cid:112) log( mKn )) .

Original: We show that for any t (cid:48) > 0 , we have D ( t (cid:48) )
Revised: We show that for any t (cid:48) > 0 , we have D ( t (cid:48) )

Original: ≤ 2 D 

Revised: ≤ 2 D

Original: = o (1) .
Revised: = o (1) .

Original: Observe that similar to Eq.
Revised: Observe that similar to Eq.

Original: (E.17), we have for any t ≥ 0 ,
Revised: (E.17), we have for any t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 ,r ,
Revised: 1 ,r ,

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where we used Condition 4.1 for the inequality, since (cid:104) z k (cid:48) , z k (cid:105) ≤ max( (cid:107) z k (cid:107) 22 , (cid:107)
Revised: where we used Condition 4.1 for the inequality, since (cid:104) z k (cid:48) , z k (cid:105) ≤ max( (cid:107) z k (cid:107) 22 , (cid:107)

Original: z k
Revised: z k

Original: [Table]
Revised: [Table]

Original: (cid:107) 22 ) =
Revised: (cid:107) 22 ) =

Original: Θ( σ 2 s d ) .
Revised: Θ( σ 2 s d ) .

Original: Comparing Eq.
Revised: Comparing Eq.

Original: (C.11) and Eq.
Revised: (C.11) and Eq.

Original: (F.1), we can roughly understand why this lemma is true: for any r ∈ [ m ] , the growth of (cid:104)
Revised: (F.1), we can roughly understand why this lemma is true: for any r ∈ [ m ] , the growth of (cid:104)

Original: [Table]
Revised: [Table]

Original: ν
Revised: ν

Original: [Table]
Revised: [Table]

Original: is at least a factor of 1 / ( σ 2 s d ) = ω (1) faster than that of (cid:104) w ( t
Revised: is at least a factor of 1 / ( σ 2 s d ) = ω (1) faster than that of (cid:104) w ( t

Original: [Table] [Table]
Revised: [Table] [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: z k
Revised: z k

Original: [Table]
Revised: [Table]

Original: We can compute
Revised: We can compute

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst and second lines are by triangle inequality.
Revised: where the ﬁrst and second lines are by triangle inequality.

Original: The third line is by comparing Eq.
Revised: The third line is by comparing Eq.

Original: (C.11) and Eq.
Revised: (C.11) and Eq.

Original: (F.1).
Revised: (F.1).

Original: The fourth line is by the fact that (cid:104) w ( t, 0) 1 ,r , ν (cid:105) is increasing in t for any r ∈ [ m ] .
Revised: The fourth line is by the fact that (cid:104) w ( t, 0) 1 ,r , ν (cid:105) is increasing in t for any r ∈ [ m ] .

Original: The sixth line is by deﬁnition of Λ ( t (cid:48) )and deﬁnition of r (0) .
Revised: The sixth line is by deﬁnition of Λ ( t (cid:48) )and deﬁnition of r (0) .

Original: The last line is by Lemma D.2.
Revised: The last line is by Lemma D.2.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where we used Λ ( t ) j is increasing in t to get the second inequality.
Revised: where we used Λ ( t ) j is increasing in t to get the second inequality.

Original: The third inequality is by direct calculation and comparison: D 0 =
Revised: The third inequality is by direct calculation and comparison: D 0 =

Original: Θ( σ s √ d (cid:112) log( mKn )) ,
Revised: Θ( σ s √ d (cid:112) log( mKn )) ,

Original: σ 2 s d = ( σ s √ d ) 2 = σ s √ d/ polylog( d ) =
Revised: σ 2 s d = ( σ s √ d ) 2 = σ s √ d/ polylog( d ) =

Original: o (1) D 0 , σ 2 s d (cid:112) log( mKn ) =
Revised: o (1) D 0 , σ 2 s d (cid:112) log( mKn ) =

Original: σ s (cid:112) log( mKn ) / polylog( d ) =
Revised: σ s (cid:112) log( mKn ) / polylog( d ) =

Original: o (1) D 0 .
Revised: o (1) D 0 .

Original: This proves the lemma.
Revised: This proves the lemma.

Original: Lemma F.2.
Revised: Lemma F.2.