Original: H OW
Revised: H OW

Original: DOES PRETEXT - BASED SELF - SUPERVISED LEARNING WORK ? A CASE STUDY
Revised: DOES PRETEXT - BASED SELF - SUPERVISED LEARNING WORK ? A CASE STUDY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised learning (SSL) is a popular machine learning paradigm that utilizes a large amount of unlabeled data to facilitate the learning from a small number of labeled data.
Revised: Self-supervised learning (SSL) is a popular machine learning paradigm that utilizes a large amount of unlabeled data to facilitate the learning from a small number of labeled data.

Original: While SSL has achieved great success in different tasks, its theoretical understanding remains largely open.
Revised: While SSL has achieved great success in different tasks, its theoretical understanding remains largely open.

Original: In this paper, we aim to theoretically understand a kind of SSL approaches based on pretext tasks.
Revised: In this paper, we aim to theoretically understand a kind of SSL approaches based on pretext tasks.

Original: In particular, the SSL approach we consider first trains a neural network based on the unlabeled data with help of pseudolabelers.
Revised: In particular, the SSL approach we consider first trains a neural network based on the unlabeled data with help of pseudolabelers.

Original: Then it linearly probes the pre-trained network on a small amount of labeled data.
Revised: Then it linearly probes the pre-trained network on a small amount of labeled data.

Original: We prove that, under certain data and neural network models, such kind of SSL can achieve nearly zero test loss, while a neural network directly trained by supervised learning on the same amount of labeled data can only achieve constant test loss.
Revised: We prove that, under certain data and neural network models, such kind of SSL can achieve nearly zero test loss, while a neural network directly trained by supervised learning on the same amount of labeled data can only achieve constant test loss.

Original: Through this case study, we demonstrate a separation between SSL and supervised learning in terms of test loss on the same amount of labeled data
Revised: Through this case study, we demonstrate a separation between SSL and supervised learning in terms of test loss on the same amount of labeled data

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the help of human-annotated labels, supervised learning has achieved remarkable success in several computer vision tasks (Girshick et al., 2014; Long et al., 2015; Krizhevsky et al., 2012; Tran et al., 2015).
Revised: With the help of human-annotated labels, supervised learning has achieved remarkable success in several computer vision tasks (Girshick et al., 2014; Long et al., 2015; Krizhevsky et al., 2012; Tran et al., 2015).

Original: However, annotating large-scale datasets (e.g., video datasets with temporal dimensions) is time-consuming and costly.
Revised: However, annotating large-scale datasets (e.g., video datasets with temporal dimensions) is time-consuming and costly.

Original: In order to reduce the number of labels used for training while maintaining a good prediction performance, a variety of methods have been proposed.
Revised: In order to reduce the number of labels used for training while maintaining a good prediction performance, a variety of methods have been proposed.

Original: Among these methods, self-supervised learning (SSL), which leverages unlabeled data to learn feature representations, is one of the most popular and successful approaches.
Revised: Among these methods, self-supervised learning (SSL), which leverages unlabeled data to learn feature representations, is one of the most popular and successful approaches.

Original: It has been shown to achieve promising prediction accuracy in computer vision (He et al., 2020;
Revised: It has been shown to achieve promising prediction accuracy in computer vision (He et al., 2020;

Original: Goyal et al., 2019; Chen et al., 2020; Grill et al., 2020; Misra & Maaten, 2020; Caron et al., 2020), speech processing (Baevski et al., 2019;
Revised: Goyal et al., 2019; Chen et al., 2020; Grill et al., 2020; Misra & Maaten, 2020; Caron et al., 2020), speech processing (Baevski et al., 2019;

Original: Wu et al., 2020; Baevski & Mohamed, 2020), and natural language processing (NLP) (Devlin et al., 2018; Radford et al., 2018) tasks.
Revised: Wu et al., 2020; Baevski & Mohamed, 2020), and natural language processing (NLP) (Devlin et al., 2018; Radford et al., 2018) tasks.

Original: One of the popular SSL approaches is pretext-based SSL , which creates human-designed supervised learning problems to leverage natural structures in the unlabeled data.
Revised: One of the popular SSL approaches is pretext-based SSL , which creates human-designed supervised learning problems to leverage natural structures in the unlabeled data.

Original: The hope is that by solving these pretext tasks, a good representation can be learned from the unlabeled data that is useful for downstream learning tasks .
Revised: The hope is that by solving these pretext tasks, a good representation can be learned from the unlabeled data that is useful for downstream learning tasks .

Original: The typical pipeline of pretext-based SSL consists of two steps.
Revised: The typical pipeline of pretext-based SSL consists of two steps.

Original: The first step is self-supervised pre-training , in which a predefined pretext task is designed, and the intrinsic features of unlabeled data are learned by neural networks.
Revised: The first step is self-supervised pre-training , in which a predefined pretext task is designed, and the intrinsic features of unlabeled data are learned by neural networks.

Original: After the training on the pretext task, models that capture the features of unlabeled data are obtained, which can be adapted to downstream tasks with the help of a linear probe (Alain & Bengio, 2016).
Revised: After the training on the pretext task, models that capture the features of unlabeled data are obtained, which can be adapted to downstream tasks with the help of a linear probe (Alain & Bengio, 2016).

Original: One special kind of pretext tasks used in SSL is based on pseudolabelers , which generate pseudolabels for unlabeled data to construct a pretext task.
Revised: One special kind of pretext tasks used in SSL is based on pseudolabelers , which generate pseudolabels for unlabeled data to construct a pretext task.

Original: For example, in image classification, one can first train a teacher network on a small labeled dataset and use it as a pseudolabeler to generate pseudolabels for large unlabeled datasets.
Revised: For example, in image classification, one can first train a teacher network on a small labeled dataset and use it as a pseudolabeler to generate pseudolabels for large unlabeled datasets.

Original: Then we can train a student network on the combination of labeled and pseudo-labeled images (Xie et al., 2020;
Revised: Then we can train a student network on the combination of labeled and pseudo-labeled images (Xie et al., 2020;

Original: Pham et al., 2021b; Rizve et al., 2021).
Revised: Pham et al., 2021b; Rizve et al., 2021).

Original: Unfortunately, despite the remarkable success of self-training for neural networks, there has not been a satisfactory theoretical understanding of such approaches.
Revised: Unfortunately, despite the remarkable success of self-training for neural networks, there has not been a satisfactory theoretical understanding of such approaches.

Original: Some attempts have been made by several recent works to theoretically understand pretext-based
Revised: Some attempts have been made by several recent works to theoretically understand pretext-based

Original: SSL with pseudolabelers .
Revised: SSL with pseudolabelers .

Original: Oymak & Gulcu (2021) considered learning a linear classifier in the
Revised: Oymak & Gulcu (2021) considered learning a linear classifier in the

Original: Gaussian mixture model setting.
Revised: Gaussian mixture model setting.

Original: They are able to show that in the high dimensional limit, the predictors found by self-training are correlated with the Bayes-optimal predictor.
Revised: They are able to show that in the high dimensional limit, the predictors found by self-training are correlated with the Bayes-optimal predictor.

Original: Frei et al. (2022c) further proved that the self-training algorithm can provably converge to the Bayes-optimal predictor for Mixture Models.
Revised: Frei et al. (2022c) further proved that the self-training algorithm can provably converge to the Bayes-optimal predictor for Mixture Models.

Original: However, their analyses are limited to the linear classifier, and cannot explain the success of SSL methods with neural networks.
Revised: However, their analyses are limited to the linear classifier, and cannot explain the success of SSL methods with neural networks.

Original: In this paper, we attempt to theoretically explain the success of SSL through a case study of pretextbased SSL with pseudolabelers.
Revised: In this paper, we attempt to theoretically explain the success of SSL through a case study of pretextbased SSL with pseudolabelers.

Original: Specifically, we focus on a data model that contains both signal patches and noise patches, where the signal patch is correlated to the label while the noise patch is not.
Revised: Specifically, we focus on a data model that contains both signal patches and noise patches, where the signal patch is correlated to the label while the noise patch is not.

Original: We then consider learning a two-layer convolutional neural network (CNN) with self-supervised learning and provide a comprehensive analysis of the learning process in both pretext and downstream tasks.
Revised: We then consider learning a two-layer convolutional neural network (CNN) with self-supervised learning and provide a comprehensive analysis of the learning process in both pretext and downstream tasks.

Original: The contributions of our work are summarized as follows.
Revised: The contributions of our work are summarized as follows.

Original: • We theoretically show that with the help of pseudolabelers the training in the pretext task converges to a global minimum of the training loss.
Revised: • We theoretically show that with the help of pseudolabelers the training in the pretext task converges to a global minimum of the training loss.

Original: Moreover, we demonstrate that the learned feature representation is highly correlated with the true labels of the data, even though the true labels are unknown and not used during the training on the pretext task.
Revised: Moreover, we demonstrate that the learned feature representation is highly correlated with the true labels of the data, even though the true labels are unknown and not used during the training on the pretext task.

Original: • Based on our analysis of the learning process for the pretext task, we further show that when linear-probing the pre-trained model in the downstream task, the final classifier can achieve nearzero test loss and test error.
Revised: • Based on our analysis of the learning process for the pretext task, we further show that when linear-probing the pre-trained model in the downstream task, the final classifier can achieve nearzero test loss and test error.

Original: Notably, these guarantees of small test loss and error only require a very small number of labeled training data.
Revised: Notably, these guarantees of small test loss and error only require a very small number of labeled training data.

Original: • As a comparison, we show that standard supervised learning cannot learn a good classifier under the same setting.
Revised: • As a comparison, we show that standard supervised learning cannot learn a good classifier under the same setting.

Original: Specifically, we show that, even when the training process converges to a global minimum of the training loss, the learned two-layer CNN can only achieve constant level test loss.
Revised: Specifically, we show that, even when the training process converges to a global minimum of the training loss, the learned two-layer CNN can only achieve constant level test loss.

Original: This, together with the aforementioned results for self-supervised learning, demonstrates the advantage of self-supervised learning over standard supervised learning.
Revised: This, together with the aforementioned results for self-supervised learning, demonstrates the advantage of self-supervised learning over standard supervised learning.

Original: Notation.
Revised: Notation.

Original: We use lower case letters, lower case bold face letters, and upper case bold face letters to denote scalars, vectors, and matrices respectively.
Revised: We use lower case letters, lower case bold face letters, and upper case bold face letters to denote scalars, vectors, and matrices respectively.

Original: For a scalar x , we use [ x ] + to denote max { x, 0 } .
Revised: For a scalar x , we use [ x ] + to denote max { x, 0 } .

Original: For a vector v = ( v 1 , · · · , v d ) ⊤ , we denote by ∥ v ∥ 2 := (cid:0) (cid:80) di
Revised: For a vector v = ( v 1 , · · · , v d ) ⊤ , we denote by ∥ v ∥ 2 := (cid:0) (cid:80) di

Original: =1 v 2 i (cid:1) 12 its ℓ 2 norm, and use supp ( v ) :
Revised: =1 v 2 i (cid:1) 12 its ℓ 2 norm, and use supp ( v ) :

Original: = { j : v j ̸ = 0 } to denote its support.
Revised: = { j : v j ̸ = 0 } to denote its support.

Original: For two sequences { a k } and { b k } , we denote a k
Revised: For two sequences { a k } and { b k } , we denote a k

Original: = O ( b k ) if | a k | ≤ C | b k | for some absolute constant C , denote a k = Ω( b k ) if b k
Revised: = O ( b k ) if | a k | ≤ C | b k | for some absolute constant C , denote a k = Ω( b k ) if b k

Original: = O ( a k ) , and denote a k = Θ( b k ) if | a k | ≤ C | b k | and a k = Ω( b k ) .
Revised: = O ( a k ) , and denote a k = Θ( b k ) if | a k | ≤ C | b k | and a k = Ω( b k ) .

Original: We also denote a k
Revised: We also denote a k

Original: = o ( b k ) if lim | a k /b k | = 0 .
Revised: = o ( b k ) if lim | a k /b k | = 0 .

Original: Finally, we use (cid:101) Θ( · ) , (cid:101) O ( · ) and (cid:101) Ω( · ) to omit logarithmic terms in the notations.
Revised: Finally, we use (cid:101) Θ( · ) , (cid:101) O ( · ) and (cid:101) Ω( · ) to omit logarithmic terms in the notations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised learning (SSL) methods in practice.
Revised: Self-supervised learning (SSL) methods in practice.

Original: Many self-supervised learning methods have been proposed to learn visual features from large-scale unlabeled images or videos without using any human annotations.
Revised: Many self-supervised learning methods have been proposed to learn visual features from large-scale unlabeled images or videos without using any human annotations.

Original: They use various pretext tasks including (1) generation-based ones such as colorizing grayscale images (Zhang et al., 2016), image inpainting (Pathak et al., 2016), image and video generation with GAN (Goodfellow et al., 2014;
Revised: They use various pretext tasks including (1) generation-based ones such as colorizing grayscale images (Zhang et al., 2016), image inpainting (Pathak et al., 2016), image and video generation with GAN (Goodfellow et al., 2014;

Original: Brock et al., 2018; Karras et al., 2019;
Revised: Brock et al., 2018; Karras et al., 2019;

Original: Vondrick et al., 2016; Tulyakov et al., 2018), and (2) context-based ones such as image jigsaw puzzle (Noroozi & Favaro, 2016), geometric transformation (Gidaris et al., 2018;
Revised: Vondrick et al., 2016; Tulyakov et al., 2018), and (2) context-based ones such as image jigsaw puzzle (Noroozi & Favaro, 2016), geometric transformation (Gidaris et al., 2018;

Original: Jing et al., 2018), frame order verification and recognition (Lee et al., 2017; Misra et al., 2016;
Revised: Jing et al., 2018), frame order verification and recognition (Lee et al., 2017; Misra et al., 2016;

Original: Wei et al., 2018).
Revised: Wei et al., 2018).

Original: Besides applications in computer vision, there are many other natural language processing (NLP) applications of self-supervised learning, such as BERT (Devlin et al., 2018) and its variants (Su et al., 2019;
Revised: Besides applications in computer vision, there are many other natural language processing (NLP) applications of self-supervised learning, such as BERT (Devlin et al., 2018) and its variants (Su et al., 2019;

Original: Yang et al., 2019; Kong et al., 2019), as well as GPT (Radford et al., 2018), all of which create auxiliary tasks for next word predictions.
Revised: Yang et al., 2019; Kong et al., 2019), as well as GPT (Radford et al., 2018), all of which create auxiliary tasks for next word predictions.

Original: Theory of SSL.
Revised: Theory of SSL.

Original: Recently, there is a line of theoretical work (Saunshi et al., 2019;
Revised: Recently, there is a line of theoretical work (Saunshi et al., 2019;

Original: Tsai et al., 2020;
Revised: Tsai et al., 2020;

Original: Mitrovic et al., 2020; Tian et al., 2020b; Wang & Isola, 2020; Tosh et al., 2021b;a) towards understanding contrastive learning , which is another mostly used SSL approach based on data augmentation.
Revised: Mitrovic et al., 2020; Tian et al., 2020b; Wang & Isola, 2020; Tosh et al., 2021b;a) towards understanding contrastive learning , which is another mostly used SSL approach based on data augmentation.

Original: Contrastive learning aims at grouping semantically similar examples, usually generated by augmentations (Chen et al., 2020; Tian et al., 2020a), together, and pushing dissimilar examples far from each other, by optimizing the so-called contrastive loss (Hadsell et al., 2006; Tian et al., 2020b).
Revised: Contrastive learning aims at grouping semantically similar examples, usually generated by augmentations (Chen et al., 2020; Tian et al., 2020a), together, and pushing dissimilar examples far from each other, by optimizing the so-called contrastive loss (Hadsell et al., 2006; Tian et al., 2020b).

Original: Saunshi et al.
Revised: Saunshi et al.

Original: (2019) first presented a theoretical framework for analyzing contrastive learning algorithms by introducing latent classes, which characterize semantically similar objects.
Revised: (2019) first presented a theoretical framework for analyzing contrastive learning algorithms by introducing latent classes, which characterize semantically similar objects.

Original: Tosh
Revised: Tosh

Original: [Figure]
Revised: [Figure]

Original: et al.
Revised: et al.

Original: (2021b;a) provided theoretical analyses of contrastive learning in both document classification with topic modeling setting and multi-view setting.
Revised: (2021b;a) provided theoretical analyses of contrastive learning in both document classification with topic modeling setting and multi-view setting.

Original: They showed that contrastive learning is capable of recovering a representation, which reveals useful information to downstream linear models and are nearly optimal.
Revised: They showed that contrastive learning is capable of recovering a representation, which reveals useful information to downstream linear models and are nearly optimal.

Original: HaoChen et al.
Revised: HaoChen et al.

Original: (2021) analyzed contrastive learning by using a novel concept called augmentation graph on data and proposed a new contrastive loss, which leads to features with provable guarantees under linear probe evaluation.
Revised: (2021) analyzed contrastive learning by using a novel concept called augmentation graph on data and proposed a new contrastive loss, which leads to features with provable guarantees under linear probe evaluation.

Original: Wen & Li (2021) analyzed the feature learning process of contrastive learning for neural network and the effect of data augmentations.
Revised: Wen & Li (2021) analyzed the feature learning process of contrastive learning for neural network and the effect of data augmentations.

Original: Saunshi et al.
Revised: Saunshi et al.

Original: (2022) demonstrated that existing analyses ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning.
Revised: (2022) demonstrated that existing analyses ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning.

Original: Unlike contrastive learning, the theoretical understanding of pretext task-based SSL is still rather limited.
Revised: Unlike contrastive learning, the theoretical understanding of pretext task-based SSL is still rather limited.

Original: The only notable works are Lee et al.
Revised: The only notable works are Lee et al.

Original: (2020) and Wei et al.
Revised: (2020) and Wei et al.

Original: Lee et al.
Revised: Lee et al.

Original: (2020) proved generalization guarantees for self-supervised algorithms using empirical risk minimizer of pretext task under certain conditional independence assumption.
Revised: (2020) proved generalization guarantees for self-supervised algorithms using empirical risk minimizer of pretext task under certain conditional independence assumption.

Original: Wei et al.
Revised: Wei et al.

Original: (2020) proved that under “expansion” assumption, the minimizers of population loss based on self-training and input-consistency regularization will achieve high prediction accuracy.
Revised: (2020) proved that under “expansion” assumption, the minimizers of population loss based on self-training and input-consistency regularization will achieve high prediction accuracy.

Original: Pseudo-Labeling.
Revised: Pseudo-Labeling.

Original: The idea of pseudo-labeling is first given by Lee et al.
Revised: The idea of pseudo-labeling is first given by Lee et al.

Original: (2013), where the authors proposed to first train the model on a batch of labeled data, then use the trained model to pseudo-label the unlabeled data, and finally use both labeled data and pseudo-labeled data to train a new model with better performance.
Revised: (2013), where the authors proposed to first train the model on a batch of labeled data, then use the trained model to pseudo-label the unlabeled data, and finally use both labeled data and pseudo-labeled data to train a new model with better performance.

Original: Recently, Xie et al.
Revised: Recently, Xie et al.

Original: Pham et al.
Revised: Pham et al.

Original: (2021a) utilized pseudo-labeling to solve semi-supervised computer vision tasks.
Revised: (2021a) utilized pseudo-labeling to solve semi-supervised computer vision tasks.

Original: Specifically, they first trained a teacher model based on the labeled ImageNet data, then used this teacher model to pseudolabel the JFT-300M dataset, and finally used the JFT-300M dataset with pseudolabels to train a student model.
Revised: Specifically, they first trained a teacher model based on the labeled ImageNet data, then used this teacher model to pseudolabel the JFT-300M dataset, and finally used the JFT-300M dataset with pseudolabels to train a student model.

Original: Feature learning by neural networks.
Revised: Feature learning by neural networks.

Original: Our work is also closely related to several recent works that study how neural networks learn the features.
Revised: Our work is also closely related to several recent works that study how neural networks learn the features.

Original: Allen-Zhu & Li (2020a) showed that adversarial training purifies the learned features by removing certain “dense mixtures” in the hidden layer weights of the network.
Revised: Allen-Zhu & Li (2020a) showed that adversarial training purifies the learned features by removing certain “dense mixtures” in the hidden layer weights of the network.

Original: Allen-Zhu & Li (2020b) studied how ensemble and knowledge distillation work in deep learning when the data have “multi-view” features.
Revised: Allen-Zhu & Li (2020b) studied how ensemble and knowledge distillation work in deep learning when the data have “multi-view” features.

Original: Zou et al.
Revised: Zou et al.

Original: (2021) studied an aspect of feature learning by Adam and GD, and showed that GD can learn the sparse features while Adam may fail even with proper regularization.
Revised: (2021) studied an aspect of feature learning by Adam and GD, and showed that GD can learn the sparse features while Adam may fail even with proper regularization.

Original: Notably, there are two concurrent works studying the benign overfitting phenomenon in learning neural networks:
Revised: Notably, there are two concurrent works studying the benign overfitting phenomenon in learning neural networks:

Original: Frei et al.
Revised: Frei et al.

Original: (2022a) established theoretical guarantees for benign overfitting of two-layer fully connected neural networks with zero training error and test error close to the Bayes-optimal error, while Cao et al.
Revised: (2022a) established theoretical guarantees for benign overfitting of two-layer fully connected neural networks with zero training error and test error close to the Bayes-optimal error, while Cao et al.

Original: (2022) studied the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN), achieving arbitrarily small training and test loss.
Revised: (2022) studied the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN), achieving arbitrarily small training and test loss.

Original: Our work studies a different aspect of feature learning afforded by self-supervised learning versus supervised learning: given a small amount of labeled data, selfsupervised learning can learn the features with the help of pretext task (i.e., pseudo-labeler), while supervised learning fails to learn the features and tends to overfit the training data.
Revised: Our work studies a different aspect of feature learning afforded by self-supervised learning versus supervised learning: given a small amount of labeled data, selfsupervised learning can learn the features with the help of pretext task (i.e., pseudo-labeler), while supervised learning fails to learn the features and tends to overfit the training data.

Original:  P ROBLEM S ETUP AND P RELIMINARIES
Revised:  P ROBLEM S ETUP AND P RELIMINARIES

Original: In this section, we first give a brief overview of the pretext-task-based self-supervised learning pipeline using pseudolabelers.
Revised: In this section, we first give a brief overview of the pretext-task-based self-supervised learning pipeline using pseudolabelers.

Original: Then we will introduce our data model, the convolutional neural network, and the detail of the training algorithms considered in this paper.
Revised: Then we will introduce our data model, the convolutional neural network, and the detail of the training algorithms considered in this paper.

Original: 3. P RETEXT - BASED S ELF - SUPERVISED L EARNING P IPELINE WITH P SEUDOLABELERS
Revised: 3. P RETEXT - BASED S ELF - SUPERVISED L EARNING P IPELINE WITH P SEUDOLABELERS

Original: In this paper, we consider a special case of pretext-based SSL (Xie et al., 2020; Pham et al., 2021b;
Revised: In this paper, we consider a special case of pretext-based SSL (Xie et al., 2020; Pham et al., 2021b;

Original: Rizve et al., 2021), which leverages pseudolabelers for pretext-training.
Revised: Rizve et al., 2021), which leverages pseudolabelers for pretext-training.

Original: The typical pipeline of this kind of pretext-based SSL is shown in Figure 1.
Revised: The typical pipeline of this kind of pretext-based SSL is shown in Figure 1.

Original: What makes it different from other pretextbased SSL methods (e.g. image jigsaw puzzle (Noroozi & Favaro, 2016), geometric transformation (Gidaris et al., 2018;
Revised: What makes it different from other pretextbased SSL methods (e.g. image jigsaw puzzle (Noroozi & Favaro, 2016), geometric transformation (Gidaris et al., 2018;

Original: Jing et al., 2018), frame order verification and recognition (Lee et al., 2017;
Revised: Jing et al., 2018), frame order verification and recognition (Lee et al., 2017;

Original: Misra et al., 2016; Wei et al., 2018)) is that it utilizes a pseudolabeler in the pre-training stage to pseudolabel unlabeled data.
Revised: Misra et al., 2016; Wei et al., 2018)) is that it utilizes a pseudolabeler in the pre-training stage to pseudolabel unlabeled data.

Original: Moreover, the case study we carry out is shown in Figure 2.
Revised: Moreover, the case study we carry out is shown in Figure 2.

Original: The pretext task trains a two-layer convolutional neural network with the help of pseudolabelers, and the downstream task trains a linear probe using the pretrained models.
Revised: The pretext task trains a two-layer convolutional neural network with the help of pseudolabelers, and the downstream task trains a linear probe using the pretrained models.

Original: 3. D ATA DISTRIBUTION
Revised: 3. D ATA DISTRIBUTION

Original: Inspired by recent work (Allen-Zhu & Li, 2020b
Revised: Inspired by recent work (Allen-Zhu & Li, 2020b

Original: ; Zou et al., 2021; Shen et al., 2022; Cao et al., 2022), we consider a data model where each data input x consists of two patches x (1) and x (2) , where each patch has d dimensions.
Revised: ; Zou et al., 2021; Shen et al., 2022; Cao et al., 2022), we consider a data model where each data input x consists of two patches x (1) and x (2) , where each patch has d dimensions.

Original: We focus on binary classification task, and present our data distribution D in the following definition.
Revised: We focus on binary classification task, and present our data distribution D in the following definition.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: Each data point ( x , y ) with x = [ x (1) ⊤ , x (2) ⊤ ] ⊤ ∈ R 2 d and y ∈ {− 1 , +1 } is generated as follows: the label y is generated as a Rademacher random variable; one of x (1) , x (2) is given by the feature vector y · v , the other is given by a noise vector ξ that is generated from a ddimensional Gaussian distribution N (cid:0) 0 , σ 2 p ( I − vv ⊤ / ∥ v ∥ 22 ) (cid:1) .
Revised: Each data point ( x , y ) with x = [ x (1) ⊤ , x (2) ⊤ ] ⊤ ∈ R 2 d and y ∈ {− 1 , +1 } is generated as follows: the label y is generated as a Rademacher random variable; one of x (1) , x (2) is given by the feature vector y · v , the other is given by a noise vector ξ that is generated from a ddimensional Gaussian distribution N (cid:0) 0 , σ 2 p ( I − vv ⊤ / ∥ v ∥ 22 ) (cid:1) .

Original: We denote by D the joint distribution of ( x , y ) , and denote by D x the marginal distribution of x .
Revised: We denote by D the joint distribution of ( x , y ) , and denote by D x the marginal distribution of x .

Original: The most natural way to think of our data model is to treat patches x (1) and x (2) as the embedding of the image data: one of them is a signal which is label-dependent, and the other one is the noise that is label-independent.
Revised: The most natural way to think of our data model is to treat patches x (1) and x (2) as the embedding of the image data: one of them is a signal which is label-dependent, and the other one is the noise that is label-independent.

Original: For simplicity, we assume that the noise patch is generated from the Gaussian distribution N ( 0 , σ 2 p · ( I − vv ⊤ · ∥ v ∥ − 2 2 )) to ensure that the noise vector is orthogonal to the signal vector v , and only consider the case where the data consists of one signal patch and one noise patch.
Revised: For simplicity, we assume that the noise patch is generated from the Gaussian distribution N ( 0 , σ 2 p · ( I − vv ⊤ · ∥ v ∥ − 2 2 )) to ensure that the noise vector is orthogonal to the signal vector v , and only consider the case where the data consists of one signal patch and one noise patch.

Original: However, our results and proof techniques can be easily extended to cover the setting with non-orthogonal signal/noise and multiple signal/noise patches.
Revised: However, our results and proof techniques can be easily extended to cover the setting with non-orthogonal signal/noise and multiple signal/noise patches.

Original: With this simple data model, in this case study we aim to show the effectiveness of self-supervised learning and explain the mechanism behind self-supervised learning with neural networks.
Revised: With this simple data model, in this case study we aim to show the effectiveness of self-supervised learning and explain the mechanism behind self-supervised learning with neural networks.

Original: Since the positions of signal and noise are not specified in Definition 3.1.
Revised: Since the positions of signal and noise are not specified in Definition 3.1.

Original: It is natural to use a classifier with a convolutional structure that applies the same function to each patch.
Revised: It is natural to use a classifier with a convolutional structure that applies the same function to each patch.

Original: More specifically, we consider learning a CNN with n l labeled examples S ′
Revised: More specifically, we consider learning a CNN with n l labeled examples S ′

Original: = { ( x ′ i , y ′ i ) }
Revised: = { ( x ′ i , y ′ i ) }

Original: n l i =1 generated from the distribution D and n u unlabeled examples S
Revised: n l i =1 generated from the distribution D and n u unlabeled examples S

Original: =
Revised: =

Original: { x i } n u i =1 generated from the marginal distribution D x , where n l is significantly smaller than the dimension d .
Revised: { x i } n u i =1 generated from the marginal distribution D x , where n l is significantly smaller than the dimension d .

Original: If we only use the labeled data, the CNN can easily overfit the training dataset by memorizing the noise patches ξ i .
Revised: If we only use the labeled data, the CNN can easily overfit the training dataset by memorizing the noise patches ξ i .

Original: Consequently, the CNN will perform badly on the fresh test data.
Revised: Consequently, the CNN will perform badly on the fresh test data.

Original: Therefore, our case is hard to learn without using unlabeled examples.
Revised: Therefore, our case is hard to learn without using unlabeled examples.

Original: 3. S UPERVISED L EARNING M ODELS
Revised: 3. S UPERVISED L EARNING M ODELS

Original: For supervised learning, we consider a two-layer CNN whose filters are applied to the patches x (1) and x (2) respectively and parameters in the second layers are set to be ± 1 .
Revised: For supervised learning, we consider a two-layer CNN whose filters are applied to the patches x (1) and x (2) respectively and parameters in the second layers are set to be ± 1 .

Original: Then the CNN can be written as f W ( x ) =
Revised: Then the CNN can be written as f W ( x ) =

Original: f +1 W ( x ) − f − 1 W ( x ) where f W ( x ) +1 , f W ( x ) − 1 are formulated as
Revised: f +1 W ( x ) − f − 1 W ( x ) where f W ( x ) +1 , f W ( x ) − 1 are formulated as

Original: [Equation]
Revised: [Equation]

Original: Here σ is activation function ReLU q ( · ) =
Revised: Here σ is activation function ReLU q ( · ) =

Original: [ · ] q + ( q > 2) , m is the width of the network, w j ∈ R d denotes the j -th filter, and W is the collection of all filters { w j } 2 mj =1 .
Revised: [ · ] q + ( q > 2) , m is the width of the network, w j ∈ R d denotes the j -th filter, and W is the collection of all filters { w j } 2 mj =1 .

Original: Given labeled training dataset
Revised: Given labeled training dataset

Original: S ′
Revised: S ′

Original: = { ( x ′ i , y ′ i ) } n l i =1 , we train the CNN model by minimizing the empirical cross-entropy loss
Revised: = { ( x ′ i , y ′ i ) } n l i =1 , we train the CNN model by minimizing the empirical cross-entropy loss

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: ( , ො𝑦
Revised: ( , ො𝑦

Original: [Figure]
Revised: [Figure]

Original: ) ( , ො𝑦
Revised: ) ( , ො𝑦

Original: [Figure]
Revised: [Figure]

Original: ) ⁞ ( , ො𝑦
Revised: ) ⁞ ( , ො𝑦

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: where L i ( W ) = ℓ (cid:0) y ′ i · f W ( x ′ i ) (cid:1) with ℓ ( z )
Revised: where L i ( W ) = ℓ (cid:0) y ′ i · f W ( x ′ i ) (cid:1) with ℓ ( z )

Original: =
Revised: =

Original: log(1 + exp( − z )) denotes the individual loss for the training example ( x i , y i ) .
Revised: log(1 + exp( − z )) denotes the individual loss for the training example ( x i , y i ) .

Original: We minimize the empirical function L S ′
Revised: We minimize the empirical function L S ′

Original: ( W ) with gradient descent as follows
Revised: ( W ) with gradient descent as follows

Original: [Equation]
Revised: [Equation]

Original: , where η > 0 is the learning rate and σ 0 defines the scale of random initialization.
Revised: , where η > 0 is the learning rate and σ 0 defines the scale of random initialization.

Original: 3. S ELF - SUPERVISED L EARNING M ODELS
Revised: 3. S ELF - SUPERVISED L EARNING M ODELS

Original: For self-supervised pre-training, we assume that we have access to K pseudolabelers { f w k } Kk =1 .
Revised: For self-supervised pre-training, we assume that we have access to K pseudolabelers { f w k } Kk =1 .

Original: The accuracy of k -th pseudolabeler is p k ∈ (1 / 2 , 1) .
Revised: The accuracy of k -th pseudolabeler is p k ∈ (1 / 2 , 1) .

Original: Then we use K pseudolabelers to generate K pseudo-labeled dataset { S k } Kk =1 , where S k := (cid:8) ( x i , (cid:98) y k,i ) (cid:12)(cid:12) (cid:98) y k,i =
Revised: Then we use K pseudolabelers to generate K pseudo-labeled dataset { S k } Kk =1 , where S k := (cid:8) ( x i , (cid:98) y k,i ) (cid:12)(cid:12) (cid:98) y k,i =

Original: f w k ( x i ) (cid:9) n u i =1 .
Revised: f w k ( x i ) (cid:9) n u i =1 .

Original: Next we solve K pretext tasks with two-layer CNN models { f W k } Kk =1 defined in (3.1) using { S k } Kk =1 respectively.
Revised: Next we solve K pretext tasks with two-layer CNN models { f W k } Kk =1 defined in (3.1) using { S k } Kk =1 respectively.

Original: Note that our result can cover K = 1 as a special case.
Revised: Note that our result can cover K = 1 as a special case.

Original: We consider learning the model parameter W k by optimizing the empirical loss of both pseudolabeled dataset S k and labeled dataset S ′
Revised: We consider learning the model parameter W k by optimizing the empirical loss of both pseudolabeled dataset S k and labeled dataset S ′

Original: = { ( x ′ i , y ′ i ) } n l i =1 with weight decay regularization
Revised: = { ( x ′ i , y ′ i ) } n l i =1 with weight decay regularization

Original: [Equation]
Revised: [Equation]

Original: where λ ≥ 0 is the regularization parameter, L i ( W k ) =
Revised: where λ ≥ 0 is the regularization parameter, L i ( W k ) =

Original: ℓ (cid:0)(cid:98)
Revised: ℓ (cid:0)(cid:98)

Original: y k,i · f W k ( x i ) (cid:1) denotes the individual loss for the pseudo-labeled data ( x i , (cid:98)
Revised: y k,i · f W k ( x i ) (cid:1) denotes the individual loss for the pseudo-labeled data ( x i , (cid:98)

Original: y k,i ) , L i ′ ( W k ) =
Revised: y k,i ) , L i ′ ( W k ) =

Original: ℓ ( y ′ i · f W k ( x ′ i )) denotes the individual loss for the labeled data ( x ′ i , y ′ i ) .
Revised: ℓ ( y ′ i · f W k ( x ′ i )) denotes the individual loss for the labeled data ( x ′ i , y ′ i ) .

Original: Our result can cover n l = 0 , which corresponds to the case that there is no labeled data during pretraining.
Revised: Our result can cover n l = 0 , which corresponds to the case that there is no labeled data during pretraining.

Original: We use gradient descent to minimize the regularized loss function L S k ∪ S ′
Revised: We use gradient descent to minimize the regularized loss function L S k ∪ S ′

Original: ( W k ) .
Revised: ( W k ) .

Original: Starting from initial W (0) k := { w
Revised: Starting from initial W (0) k := { w

Original: [Equation]
Revised: [Equation]

Original: 0) k,j , j ∈ [2 m ] } , gradient descent update rule is as follows
Revised: 0) k,j , j ∈ [2 m ] } , gradient descent update rule is as follows

Original: [Equation]
Revised: [Equation]

Original: where η > 0 is the learning rate and σ 0 defines the scale of random initialization.
Revised: where η > 0 is the learning rate and σ 0 defines the scale of random initialization.

Original: • Downstream Task: Linear Model.
Revised: • Downstream Task: Linear Model.

Original: The training of pretext task gives us K CNN models with parameters { W ∗ k } Kk =1 .
Revised: The training of pretext task gives us K CNN models with parameters { W ∗ k } Kk =1 .

Original: Based on them, for the downstream task, we consider a linear model
Revised: Based on them, for the downstream task, we consider a linear model

Original: [Equation]
Revised: [Equation]

Original: where a k ∈ R denotes the trainable weight for the k -th pretext model.
Revised: where a k ∈ R denotes the trainable weight for the k -th pretext model.

Original: Then, given { f W ∗ k } K k = and labeled training data S ′ = { ( x ′ i , y ′ i ) } ni =1 , we consider learning the downstream linear model parameter a by optimizing the following empirical loss
Revised: Then, given { f W ∗ k } K k = and labeled training data S ′ = { ( x ′ i , y ′ i ) } ni =1 , we consider learning the downstream linear model parameter a by optimizing the following empirical loss

Original: [Equation]
Revised: [Equation]

Original: We initialize a as an all zero vector and optimize the empirical loss by gradient descent with learning rate η , i.e.,
Revised: We initialize a as an all zero vector and optimize the empirical loss by gradient descent with learning rate η , i.e.,

Original: [Equation]
Revised: [Equation]

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: In this section, we present the main theoretical results in this paper.
Revised: In this section, we present the main theoretical results in this paper.

Original: We start with a condition that is required by our analysis.
Revised: We start with a condition that is required by our analysis.

Original: Condition 4.1.
Revised: Condition 4.1.

Original: The strength of the signal is ∥ v ∥
Revised: The strength of the signal is ∥ v ∥

Original: 22 = Θ( d ) , the noise variance is σ p =
Revised: 22 = Θ( d ) , the noise variance is σ p =

Original: Θ( d ϵ ) , where 0 < ϵ < 1 / 8 is a small constant, and the width of the network satisfies m
Revised: Θ( d ϵ ) , where 0 < ϵ < 1 / 8 is a small constant, and the width of the network satisfies m

Original: = polylog ( d ) .
Revised: = polylog ( d ) .

Original: We also assume that the size of the unlabeled dataset n u = Ω( d 4 ϵ ) , and labeled data n l = (cid:101) Θ(1) .
Revised: We also assume that the size of the unlabeled dataset n u = Ω( d 4 ϵ ) , and labeled data n l = (cid:101) Θ(1) .

Original: For both supervise learning and self-supervised learning settings, we initialize the weight with σ 0 =
Revised: For both supervise learning and self-supervised learning settings, we initialize the weight with σ 0 =

Original: Θ( d − 3 / 4 ) .
Revised: Θ( d − 3 / 4 ) .

Original: For self-supervised learning, we require λ
Revised: For self-supervised learning, we require λ

Original: = o ( d 3 / 4 ) and assume that there exists a constant C such that for all pseudolabelers, their test accuracy p k > 1 / 2 + C .
Revised: = o ( d 3 / 4 ) and assume that there exists a constant C such that for all pseudolabelers, their test accuracy p k > 1 / 2 + C .

Original: Since we generate the noise patch from the Gaussian distribution, the strength of the noise patch is ∥ ξ ∥ 2 2 ≈ d 1+ ϵ by standard concentration inequalities, which is larger than the strength of the signal patch ∥ v ∥ 2 2 =
Revised: Since we generate the noise patch from the Gaussian distribution, the strength of the noise patch is ∥ ξ ∥ 2 2 ≈ d 1+ ϵ by standard concentration inequalities, which is larger than the strength of the signal patch ∥ v ∥ 2 2 =

Original: Θ( d ) .
Revised: Θ( d ) .

Original: Therefore, Condition 4.1 defines a setting with large noises.
Revised: Therefore, Condition 4.1 defines a setting with large noises.

Original: The condition of d ≫ n u ≫ n l further ensures that learning is in a sufficiently over-parameterized setting.
Revised: The condition of d ≫ n u ≫ n l further ensures that learning is in a sufficiently over-parameterized setting.

Original: Here we only require the neural network width m to be polylogarithmic in the dimension d and require the psudolablers to perform better than a random guess.
Revised: Here we only require the neural network width m to be polylogarithmic in the dimension d and require the psudolablers to perform better than a random guess.

Original: Theorem 4.2 (Self-supervised Learning: Pretext) .
Revised: Theorem 4.2 (Self-supervised Learning: Pretext) .

Original: Let k ∈
Revised: Let k ∈

Original: [ K ] and consider the self-supervised pre-training of f W k ( x ) .
Revised: [ K ] and consider the self-supervised pre-training of f W k ( x ) .

Original: For any test data point ( x , y ) , denote (cid:98) y
Revised: For any test data point ( x , y ) , denote (cid:98) y

Original: =
Revised: =

Original: f w k ( x ) .
Revised: f w k ( x ) .

Original: Then under Condition 4.1, after T 0 = (cid:101) Θ( d − 34 η − 1 ) training iterations with learning rate
Revised: Then under Condition 4.1, after T 0 = (cid:101) Θ( d − 34 η − 1 ) training iterations with learning rate

Original: η = O ( d − 1 . 1 ) , the trained neural network can achieve nearly 0 test error on the distribution D : P ( x ,y )
Revised: η = O ( d − 1 . 1 ) , the trained neural network can achieve nearly 0 test error on the distribution D : P ( x ,y )

Original: ∼D [ y · f W ( T 0)
Revised: ∼D [ y · f W ( T 0)

Original: k ( x ) ≤ 0]
Revised: k ( x ) ≤ 0]

Original: = o (1) .
Revised: = o (1) .

Original: Theorem 4.2 characterizes the prediction power of the feature representation learned in the pretext task using unlabeled data.
Revised: Theorem 4.2 characterizes the prediction power of the feature representation learned in the pretext task using unlabeled data.

Original: For any test data point ( x , y ) , the sign of y can be predicted based on f W ( T 0)
Revised: For any test data point ( x , y ) , the sign of y can be predicted based on f W ( T 0)

Original: ( x ) with high probability.
Revised: ( x ) with high probability.

Original: Theorem 4.3 (Self-supervised Learning: Downstream) .
Revised: Theorem 4.3 (Self-supervised Learning: Downstream) .

Original: Let (cid:8) f W
Revised: Let (cid:8) f W

Original: ( Tk 0 )
Revised: ( Tk 0 )

Original: k (cid:9)
Revised: k (cid:9)

Original: d k =1 be the neural networks trained according to the d pretext tasks, and consider the learning of the downstream task based in (cid:8) f W
Revised: d k =1 be the neural networks trained according to the d pretext tasks, and consider the learning of the downstream task based in (cid:8) f W

Original: ( Tk 0 )
Revised: ( Tk 0 )

Original: k (cid:9)
Revised: k (cid:9)

Original: d k =1 .
Revised: d k =1 .

Original: Under Condition 4.1, after T ′ =
Revised: Under Condition 4.1, after T ′ =

Original: Θ( d 0 . 1 /η ) iterations with learning rate η = Θ(1) , with probability 1 − o (1) , the obtained a ( T ′ ) satisfies:
Revised: Θ( d 0 . 1 /η ) iterations with learning rate η = Θ(1) , with probability 1 − o (1) , the obtained a ( T ′ ) satisfies:

Original: • Training error is 0: 1 n (cid:80) ni
Revised: • Training error is 0: 1 n (cid:80) ni

Original: =1 

Revised: =1

Original: [ y i · g a ( T ′ )
Revised: [ y i · g a ( T ′ )

Original: ( x i ) ≤ 0] = 0 .
Revised: ( x i ) ≤ 0] = 0 .

Original: • Test error and loss are nearly 0: P ( x ,y )
Revised: • Test error and loss are nearly 0: P ( x ,y )

Original: ∼D [ y · g a ( T ′ )
Revised: ∼D [ y · g a ( T ′ )

Original: ( x ) ≤ 0]
Revised: ( x ) ≤ 0]

Original: = o (1) , L D (cid:0) a ( T ′ ) (cid:1) =
Revised: = o (1) , L D (cid:0) a ( T ′ ) (cid:1) =

Original: o (1) .
Revised: o (1) .

Original: Theorem 4.3 shows that the feature representation learned based on the unsupervised pretext tasks can ensure small training and test errors for the supervised downstream task.
Revised: Theorem 4.3 shows that the feature representation learned based on the unsupervised pretext tasks can ensure small training and test errors for the supervised downstream task.

Original: Notably, this result holds even though we assume that there are only a constant number of labeled data.
Revised: Notably, this result holds even though we assume that there are only a constant number of labeled data.

Original: This shows that self-supervised learning can significantly reduce the need for a large labeled training dataset.
Revised: This shows that self-supervised learning can significantly reduce the need for a large labeled training dataset.

Original: For comparison, we also have the following guarantees on the performance of standard supervised learning of CNNs.
Revised: For comparison, we also have the following guarantees on the performance of standard supervised learning of CNNs.

Original: Theorem 4.4 (Supervised Learning) .
Revised: Theorem 4.4 (Supervised Learning) .

Original: Under supervised learning setting, after gradient descent for
Revised: Under supervised learning setting, after gradient descent for

Original: T = (cid:101)
Revised: T = (cid:101)

Original: Θ( d (1 / 4 − ϵ ) q − 3 / 2 η − 1 ) iterations with learning rate η
Revised: Θ( d (1 / 4 − ϵ ) q − 3 / 2 η − 1 ) iterations with learning rate η

Original: = O ( d − 1 − 2 ϵ ) , then there exists t ≤ T such that with probability 1 − o (1) the CNNs defined in (3.1) with parameter W ( t ) satisfies:
Revised: = O ( d − 1 − 2 ϵ ) , then there exists t ≤ T such that with probability 1 − o (1) the CNNs defined in (3.1) with parameter W ( t ) satisfies:

Original: • Training loss is nearly zero: L S ′ (cid:0) W ( t ) (cid:1)
Revised: • Training loss is nearly zero: L S ′ (cid:0) W ( t ) (cid:1)

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: • Test loss is high: L D (cid:0) W ( t ) (cid:1) = Θ(1) .
Revised: • Test loss is high: L D (cid:0) W ( t ) (cid:1) = Θ(1) .

Original: Theorem 4.4 shows that although standard supervised learning can train a CNN model with nearly zero training loss, the obtained CNN model generalizes poorly to test data.
Revised: Theorem 4.4 shows that although standard supervised learning can train a CNN model with nearly zero training loss, the obtained CNN model generalizes poorly to test data.

Original: Comparing Theorem 4. with Theorem 4.3 shows that the generalization of self-supervised learning and supervised learning are largely different.
Revised: Comparing Theorem 4. with Theorem 4.3 shows that the generalization of self-supervised learning and supervised learning are largely different.

Original: The reason behind this difference is that, the pretext task, with a relatively large number of unlabeled training data, helps learn a feature representation that captures the feature v in our data model, while direct application of supervised learning can only memorize the noises
Revised: The reason behind this difference is that, the pretext task, with a relatively large number of unlabeled training data, helps learn a feature representation that captures the feature v in our data model, while direct application of supervised learning can only memorize the noises

Original: ξ ′ i , i ∈ [ n l ] in the training dataset, which is independent of the labels of the data.
Revised: ξ ′ i , i ∈ [ n l ] in the training dataset, which is independent of the labels of the data.

Original: A recent line of work (Oymak & Gulcu, 2021; Frei et al., 2022c) studies the pretext-based SSL methods with pseudolabelers.
Revised: A recent line of work (Oymak & Gulcu, 2021; Frei et al., 2022c) studies the pretext-based SSL methods with pseudolabelers.

Original: Our results are different from theirs in several aspects: (i) we are considering learning with CNNs rather than a linear model, so the problem is highly non-convex with various local minima, which makes the optimization analysis more challenging; (ii) the Bayesian optimal predictor is no longer unique for CNNs.
Revised: Our results are different from theirs in several aspects: (i) we are considering learning with CNNs rather than a linear model, so the problem is highly non-convex with various local minima, which makes the optimization analysis more challenging; (ii) the Bayesian optimal predictor is no longer unique for CNNs.

Original: Therefore, we measure the quality of the learned features via downstream task instead of making a comparison with the Bayesian optimal predictor; (iii) They can only deal with the case where the teacher network (pseudolabeler) is the same as the student network (Frei et al., 2022c) or the case where the teacher network (pseudolabeler) is at least as complex as the student network (Oymak & Gulcu, 2021).
Revised: Therefore, we measure the quality of the learned features via downstream task instead of making a comparison with the Bayesian optimal predictor; (iii) They can only deal with the case where the teacher network (pseudolabeler) is the same as the student network (Frei et al., 2022c) or the case where the teacher network (pseudolabeler) is at least as complex as the student network (Oymak & Gulcu, 2021).

Original: However, our teacher network (pseudolabeler) is not specified and can be any structure, such as a linear network.
Revised: However, our teacher network (pseudolabeler) is not specified and can be any structure, such as a linear network.

Original: Therefore we can handle the case where the student network is more complex than the teacher network, one of the most natural settings for self-training with pseudolabeler (Xie et al., 2020).
Revised: Therefore we can handle the case where the student network is more complex than the teacher network, one of the most natural settings for self-training with pseudolabeler (Xie et al., 2020).

Original:  P ROOF S KETCH
Revised:  P ROOF S KETCH

Original: In this section, we present the proof sketch for the self-supervised learning setting.
Revised: In this section, we present the proof sketch for the self-supervised learning setting.

Original: And the proof sketch of the supervised learning setting is given in the appendix.
Revised: And the proof sketch of the supervised learning setting is given in the appendix.

Original: Pretext task.
Revised: Pretext task.

Original: We consider learning K functions
Revised: We consider learning K functions

Original: f W k ( x ) , k ∈ [ K ] based on the pretext task.
Revised: f W k ( x ) , k ∈ [ K ] based on the pretext task.

Original: Since the learning process of these K functions can be analyzed in exactly the same way, here we only focus on the learning of one of these functions.
Revised: Since the learning process of these K functions can be analyzed in exactly the same way, here we only focus on the learning of one of these functions.

Original: For simplicity of notation, we drop the subscript k in the following proof sketch.
Revised: For simplicity of notation, we drop the subscript k in the following proof sketch.

Original: Our study of the pretext task focuses on two aspects of the training process: feature learning and noise memorization .
Revised: Our study of the pretext task focuses on two aspects of the training process: feature learning and noise memorization .

Original: Specifically, we aim to monitor how the filters in the CNN model learn the feature vector v and the noise vectors ξ i ’s.
Revised: Specifically, we aim to monitor how the filters in the CNN model learn the feature vector v and the noise vectors ξ i ’s.

Original: Therefore, we introduce the following notations.
Revised: Therefore, we introduce the following notations.

Original: [Equation]
Revised: [Equation]

Original: Based on the above definitions for r ∈ {± 1 } , a larger (cid:98) Λ ( t ) r implies better feature learning along the positive feature direction v , while a larger ¯Λ ( t ) r implies better feature learning along the negative feature direction − v .
Revised: Based on the above definitions for r ∈ {± 1 } , a larger (cid:98) Λ ( t ) r implies better feature learning along the positive feature direction v , while a larger ¯Λ ( t ) r implies better feature learning along the negative feature direction − v .

Original: Moreover, a larger Γ ( t ) implies a higher level of noise memorization.
Revised: Moreover, a larger Γ ( t ) implies a higher level of noise memorization.

Original: Based on the update rule of gradient descent, for the inner products ⟨ w ( t ) j , v ⟩ and ⟨ w ( t ) j , ξ l ⟩ , for j ∈ [2 m ] , l ∈ [ n u ] , we can obtain iterative equations in (A.1).
Revised: Based on the update rule of gradient descent, for the inner products ⟨ w ( t ) j , v ⟩ and ⟨ w ( t ) j , ξ l ⟩ , for j ∈ [2 m ] , l ∈ [ n u ] , we can obtain iterative equations in (A.1).

Original: With the help of the iterative equations and definitions in (5.1), we can further show the following lemma.
Revised: With the help of the iterative equations and definitions in (5.1), we can further show the following lemma.

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Assume we use both unlabeled data with pseudolabels generated by the pseudolabeler and labeled data for the training of our CNN model.
Revised: Assume we use both unlabeled data with pseudolabels generated by the pseudolabeler and labeled data for the training of our CNN model.

Original: Then for r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) , then for t ∈ [0 , T r ] , we have
Revised: Then for r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) , then for t ∈ [0 , T r ] , we have

Original: [Equation]
Revised: [Equation]

Original: where C is defined in Condition 4.1.
Revised: where C is defined in Condition 4.1.

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: Assume we use only labeled data for the training of our CNN model.
Revised: Assume we use only labeled data for the training of our CNN model.

Original: Then for i ∈ [ n l ] , let T ′
Revised: Then for i ∈ [ n l ] , let T ′

Original: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) , then we have
Revised: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) , then we have

Original: [Equation]
Revised: [Equation]

Original: Based on the results in Lemma 5.1, we can observe that if both pseudo-labeled and labeled data are used for training, the CNN will learn the positive direction of the feature vector v , while barely tending to fit the negative direction of the feature vector or memorize the noise.
Revised: Based on the results in Lemma 5.1, we can observe that if both pseudo-labeled and labeled data are used for training, the CNN will learn the positive direction of the feature vector v , while barely tending to fit the negative direction of the feature vector or memorize the noise.

Original: And if only labeled data are used, the CNN will fit noise faster than a feature, which can be seen from Lemma 5.2.
Revised: And if only labeled data are used, the CNN will fit noise faster than a feature, which can be seen from Lemma 5.2.

Original: Leveraging Lemmas 5.1 and 5.2, we can obtain following Lemmas 5.3 and 5.4, which characterize the magnitude of feature learning and noise memorization.
Revised: Leveraging Lemmas 5.1 and 5.2, we can obtain following Lemmas 5.3 and 5.4, which characterize the magnitude of feature learning and noise memorization.

Original: Lemma 5.3.
Revised: Lemma 5.3.

Original: If both pseudo-labeled and labeled data are used to train CNN, for r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) respectively.
Revised: If both pseudo-labeled and labeled data are used to train CNN, for r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) respectively.

Original: Let T 0 =
Revised: Let T 0 =

Original: max r ∈{± 1 } { T r } .
Revised: max r ∈{± 1 } { T r } .

Original: Then, it holds that (cid:98) Λ ( T 0 )
Revised: Then, it holds that (cid:98) Λ ( T 0 )

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: and Γ ( t ) =
Revised: and Γ ( t ) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: ( d − 1 4 + ϵ ) for all t ∈ [0 , T 0 ] .
Revised: ( d − 1 4 + ϵ ) for all t ∈ [0 , T 0 ] .

Original: Lemma 5.4.
Revised: Lemma 5.4.

Original: If only labeled data are used to train CNN, for i ∈ [ n l ] , let T ′
Revised: If only labeled data are used to train CNN, for i ∈ [ n l ] , let T ′

Original: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) .
Revised: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) .

Original: Let T ′ 

Revised: Let T ′

Original: = max i ∈
Revised: = max i ∈

Original: [ n l ] T ′ i .
Revised: [ n l ] T ′ i .

Original: Then, it holds that (cid:98) Λ r = (cid:101) O
Revised: Then, it holds that (cid:98) Λ r = (cid:101) O

Original: ( d − 14 ) , ¯Λ r = (cid:101)
Revised: ( d − 14 ) , ¯Λ r = (cid:101)

Original: O
Revised: O

Original: ( d − 14 ) for r ∈ {± 1 } and Γ ′ ( t ) i = (cid:101) Θ(1) for i ∈ [ n l ] .
Revised: ( d − 14 ) for r ∈ {± 1 } and Γ ′ ( t ) i = (cid:101) Θ(1) for i ∈ [ n l ] .

Original: The above results indicate the deviation between the two settings.
Revised: The above results indicate the deviation between the two settings.

Original: The reason is that assume we consider a sequence { x t } with iterative equation x t +1 = x t + η · C t x q − 1 t .
Revised: The reason is that assume we consider a sequence { x t } with iterative equation x t +1 = x t + η · C t x q − 1 t .

Original: If we only use labeled data, as shown in Lemma 5.2, Γ ′ ( t ) i has C t = (cid:101)
Revised: If we only use labeled data, as shown in Lemma 5.2, Γ ′ ( t ) i has C t = (cid:101)

Original: Θ( d 1+2 ϵ ) while (cid:98) Λ (
Revised: Θ( d 1+2 ϵ ) while (cid:98) Λ (

Original: t ) r has C t =
Revised: t ) r has C t =

Original: Θ( d ) , therefore Γ ′ ( t ) i increases faster than (cid:98) Λ ( t ) r .
Revised: Θ( d ) , therefore Γ ′ ( t ) i increases faster than (cid:98) Λ ( t ) r .

Original: In contrast, if we use both labeled data and pseudo-labeled data, C t will be (cid:101)
Revised: In contrast, if we use both labeled data and pseudo-labeled data, C t will be (cid:101)

Original: Θ( d 1 − 2 ϵ ) for Γ ′ ( t ) i and Θ( d ) for (cid:98) Λ ( t ) r , leading to a slower increasing speed of Γ ′ ( t ) i .
Revised: Θ( d 1 − 2 ϵ ) for Γ ′ ( t ) i and Θ( d ) for (cid:98) Λ ( t ) r , leading to a slower increasing speed of Γ ′ ( t ) i .

Original: Downstream task.
Revised: Downstream task.

Original: After the training on the pretext tasks, we have obtained K classifiers (cid:8) f W ( Tk 0 )
Revised: After the training on the pretext tasks, we have obtained K classifiers (cid:8) f W ( Tk 0 )

Original: k (cid:9) K k
Revised: k (cid:9) K k

Original: =1 .
Revised: =1 .

Original: Now we train the second-layer parameters a with the training data whose true labels are available.
Revised: Now we train the second-layer parameters a with the training data whose true labels are available.

Original: The following lemma shows that the l 1 -norm of a will increase with a logarithmic order.
Revised: The following lemma shows that the l 1 -norm of a will increase with a logarithmic order.

Original: Lemma 5.5.
Revised: Lemma 5.5.

Original: For any learning rate η = Θ(1) , we have (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = log( t ) / (cid:101) Θ(1) .
Revised: For any learning rate η = Θ(1) , we have (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = log( t ) / (cid:101) Θ(1) .

Original: For any labeled data ( x ′ i , y ′ i ) ∈ S ′ , we have with high probability that y ′ i · f W ( t ) ( x ′ i ) =
Revised: For any labeled data ( x ′ i , y ′ i ) ∈ S ′ , we have with high probability that y ′ i · f W ( t ) ( x ′ i ) =

Original: (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 · (cid:101) Θ(1) .
Revised: (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 · (cid:101) Θ(1) .

Original: For any newly generated data ( x , y ) ∼ D , we also have with high probability that y · f W ( t ) ( x ) =
Revised: For any newly generated data ( x , y ) ∼ D , we also have with high probability that y · f W ( t ) ( x ) =

Original: (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 · (cid:101) Θ(1) .
Revised: (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 · (cid:101) Θ(1) .

Original: With the help of the above lemma and note that training error and test error are related to y · f W ( T 0) ( x ) and test loss is related to ∥ a ( T 0 ) ∥ 1 , we can prove that after T ′ =
Revised: With the help of the above lemma and note that training error and test error are related to y · f W ( T 0) ( x ) and test loss is related to ∥ a ( T 0 ) ∥ 1 , we can prove that after T ′ =

Original: Θ( d 0 . 01 ) iterations with learning rate η = Θ(1) , the model can achieve nearly zero training error, test error, training loss and test loss.
Revised: Θ( d 0 . 01 ) iterations with learning rate η = Θ(1) , the model can achieve nearly zero training error, test error, training loss and test loss.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we perform numerical experiments on synthetic datasets, generated according to
Revised: In this section, we perform numerical experiments on synthetic datasets, generated according to

Original: Definition 3.1, to verify our main theoretical results.
Revised: Definition 3.1, to verify our main theoretical results.

Original: In particular, we set the problem dimension d = 10000 , labeled training sample size n l = 20 ( 10 positive samples and 10 negative samples), pseudolabeled training sample size n u = 20000 (10000 positive samples and 10000 negative samples), feature vector v sampled from distribution N ( 0 , I ) and noise vector sampled from distribution
Revised: In particular, we set the problem dimension d = 10000 , labeled training sample size n l = 20 ( 10 positive samples and 10 negative samples), pseudolabeled training sample size n u = 20000 (10000 positive samples and 10000 negative samples), feature vector v sampled from distribution N ( 0 , I ) and noise vector sampled from distribution

Original: N ( 0 , σ 2 p I ) where σ p = 10 d 0 . 01 .
Revised: N ( 0 , σ 2 p I ) where σ p = 10 d 0 . 01 .

Original: For self-supervised learning task, we have a linear pseudolabeler with test error 0 .
Revised: For self-supervised learning task, we have a linear pseudolabeler with test error 0 .

Original: 196 ± 0 . 044 .
Revised: 196 ± 0 . 044 .

Original: Then, we use this classifier to generate pseudolabels for n u = 20000 unlabeled samples in order to help self-supervised learning.
Revised: Then, we use this classifier to generate pseudolabels for n u = 20000 unlabeled samples in order to help self-supervised learning.

Original: After that, for pretext task, we use these pseudo-labeled samples and n l labeled samples together to train a CNN with network width m = 20 , activation function σ ( z ) =
Revised: After that, for pretext task, we use these pseudo-labeled samples and n l labeled samples together to train a CNN with network width m = 20 , activation function σ ( z ) =

Original: [ z ] 3+ , regularization parameter λ = 0 .
Revised: [ z ] 3+ , regularization parameter λ = 0 .

Original: 1 and learning rate η = 1 × 10 − 4 .
Revised: 1 and learning rate η = 1 × 10 − 4 .

Original: Besides, we initialize
Revised: Besides, we initialize

Original: CNN parameters from N (0 , σ 20 ) , where σ 0 = 0 .
Revised: CNN parameters from N (0 , σ 20 ) , where σ 0 = 0 .

Original: 1 × d − 3 / 4 .
Revised: 1 × d − 3 / 4 .

Original: After 200 iterations, we can obtain a
Revised: After 200 iterations, we can obtain a

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: CNN model with training error close to the error of pseudolabeler and zero test error, according to
Revised: CNN model with training error close to the error of pseudolabeler and zero test error, according to

Original: Table 6.
Revised: Table 6.

Original: And for a downstream task, we use n l labeled samples to train a linear probe.
Revised: And for a downstream task, we use n l labeled samples to train a linear probe.

Original: By applying learning rate η = 0 . 1 and after T = 100 iterations, we can obtain a final model with low training and test loss as well as 100% training accuracy and test accuracy.
Revised: By applying learning rate η = 0 . 1 and after T = 100 iterations, we can obtain a final model with low training and test loss as well as 100% training accuracy and test accuracy.

Original: For supervised learning task, we directly use n l labeled data to train a CNN with network width m = 20 , activation function σ ( z ) =
Revised: For supervised learning task, we directly use n l labeled data to train a CNN with network width m = 20 , activation function σ ( z ) =

Original: [ z ] 3+ , learning rate η = 1 × 10 − 4 .
Revised: [ z ] 3+ , learning rate η = 1 × 10 − 4 .

Original: After 200 iterations, we obtain a CNN with 0 training error and small training loss, about 0 .
Revised: After 200 iterations, we obtain a CNN with 0 training error and small training loss, about 0 .

Original: 5 test error and high test loss, which indicates supervised learning will give a model that behaves badly and even no better than a random guess.
Revised: 5 test error and high test loss, which indicates supervised learning will give a model that behaves badly and even no better than a random guess.

Original: Moreover, for synthetic data experiments, we also calculate the inner products max j ∈ [ m ] ⟨ w ( t ) j , v ⟩ and max j ∈ [2 m ] (cid:8) max i ∈ [ n u ] ⟨ w ( t ) j , ξ i ⟩ , max i ∈ [ n l ] ⟨ w ( t ) j , ξ ′ i ⟩ (cid:9) , i.e. (cid:98) Λ ( t ) 1 and Γ ( t ) , representing feature learning and noise memorization respectively, to verify our key lemmas.
Revised: Moreover, for synthetic data experiments, we also calculate the inner products max j ∈ [ m ] ⟨ w ( t ) j , v ⟩ and max j ∈ [2 m ] (cid:8) max i ∈ [ n u ] ⟨ w ( t ) j , ξ i ⟩ , max i ∈ [ n l ] ⟨ w ( t ) j , ξ ′ i ⟩ (cid:9) , i.e. (cid:98) Λ ( t ) 1 and Γ ( t ) , representing feature learning and noise memorization respectively, to verify our key lemmas.

Original: The results are reported in
Revised: The results are reported in

Original: Figure 3.
Revised: Figure 3.

Original: It can be seen from Figure 3 that under self-supervised learning setting the algorithm will the feature learning will dominate the noise memorization though the noise patch has a larger norm than the signal patch, while under the supervised learning setting, the algorithm will entirely forget the feature but fit noise.
Revised: It can be seen from Figure 3 that under self-supervised learning setting the algorithm will the feature learning will dominate the noise memorization though the noise patch has a larger norm than the signal patch, while under the supervised learning setting, the algorithm will entirely forget the feature but fit noise.

Original: This verifies Lemmas 5.3 and 5.4.
Revised: This verifies Lemmas 5.3 and 5.4.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we consider a special kind of pretext-based self-supervised learning with pseudolabelers and provide a theoretical understanding towards the success of self-supervised learning as well as the advantage of self-supervised learning over supervised learning through a case study.
Revised: In this paper, we consider a special kind of pretext-based self-supervised learning with pseudolabelers and provide a theoretical understanding towards the success of self-supervised learning as well as the advantage of self-supervised learning over supervised learning through a case study.

Original: By considering a simple data model and two-layer CNN, we present a comprehensive analysis of the training procedure from a beyond-NTK feature learning perspective.
Revised: By considering a simple data model and two-layer CNN, we present a comprehensive analysis of the training procedure from a beyond-NTK feature learning perspective.

Original: We prove that the final classifier of a self-supervised learning scenario can achieve near-zero test loss and error with only a small number of labeled training data, while its supervised-learned counterpart fails to achieve the same performance with the same data complexity.
Revised: We prove that the final classifier of a self-supervised learning scenario can achieve near-zero test loss and error with only a small number of labeled training data, while its supervised-learned counterpart fails to achieve the same performance with the same data complexity.

Original: In the current paper, we only focus on the simplest possible data and neural network models to study self-supervised learning.
Revised: In the current paper, we only focus on the simplest possible data and neural network models to study self-supervised learning.

Original: For example, the second layer of the CNN is fixed during the training.
Revised: For example, the second layer of the CNN is fixed during the training.

Original: What if the second-layer is trainable?
Revised: What if the second-layer is trainable?

Original: In addition, the stride is the same as the filter size in the current CNN, and it is reasonable to have the stride to be smaller than the filter size.
Revised: In addition, the stride is the same as the filter size in the current CNN, and it is reasonable to have the stride to be smaller than the filter size.

Original: On the other hand, it would be interesting to consider linearly non-separable data (Shi et al., 2022;
Revised: On the other hand, it would be interesting to consider linearly non-separable data (Shi et al., 2022;

Original: Frei et al., 2022b; Damian et al., 2022) and ReLU activation function with the help of pre-activation noise (Allen-Zhu &
Revised: Frei et al., 2022b; Damian et al., 2022) and ReLU activation function with the help of pre-activation noise (Allen-Zhu &

Original: A P ROOF FOR S ELF - SUPERVISED L EARNING S ETTING
Revised: A P ROOF FOR S ELF - SUPERVISED L EARNING S ETTING

Original: We consider learning K functions
Revised: We consider learning K functions

Original: f W k ( x ) , k ∈ [ K ] based on the pretext task.
Revised: f W k ( x ) , k ∈ [ K ] based on the pretext task.

Original: Since the learning process of these K functions can be analyzed in exactly the same way, here we only focus on the learning of one of these functions.
Revised: Since the learning process of these K functions can be analyzed in exactly the same way, here we only focus on the learning of one of these functions.

Original: For simplicity of notation, we drop the subscript k in the following proof for section A.1, A.2, A.3, A.4, A.5, A.6 and A.7.
Revised: For simplicity of notation, we drop the subscript k in the following proof for section A.1, A.2, A.3, A.4, A.5, A.6 and A.7.

Original: A. G RADIENT C ALCULATION
Revised: A. G RADIENT C ALCULATION

Original: Lemma A.1 (Gradient Calculation) .
Revised: Lemma A.1 (Gradient Calculation) .

Original: The gradient of loss function L S ( W ) with respect to weight parameters w j is
Revised: The gradient of loss function L S ( W ) with respect to weight parameters w j is

Original: [Equation]
Revised: [Equation]

Original: for 1 ≤ j ≤ m ; and
Revised: for 1 ≤ j ≤ m ; and

Original: [Equation]
Revised: [Equation]

Original: ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1)(cid:19) + λ · w j , for m + 1 ≤ j ≤ 2 m , where − ℓ ′ (cid:0)(cid:98)
Revised: ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1)(cid:19) + λ · w j , for m + 1 ≤ j ≤ 2 m , where − ℓ ′ (cid:0)(cid:98)

Original: y i · f W ( x i ) (cid:1) =
Revised: y i · f W ( x i ) (cid:1) =

Original: exp [ − (cid:98) y
Revised: exp [ − (cid:98) y

Original: i · f W ( x i )] / (1 + exp [ − (cid:98) y
Revised: i · f W ( x i )] / (1 + exp [ − (cid:98) y

Original: i · f W ( x i )]) is denoted as c i and − ℓ ′
Revised: i · f W ( x i )]) is denoted as c i and − ℓ ′

Original: ( y ′ i · f W ( x ′ i ))
Revised: ( y ′ i · f W ( x ′ i ))

Original: =
Revised: =

Original: exp[ − y ′
Revised: exp[ − y ′

Original: i
Revised: i

Original: · f W ( x ′ i )] / (1 + exp
Revised: · f W ( x ′ i )] / (1 + exp

Original: [ − y ′ i · f W ( x ′ i )]) is denoted as b i .
Revised: [ − y ′ i · f W ( x ′ i )]) is denoted as b i .

Original: Proof of Lemma A.1.
Revised: Proof of Lemma A.1.

Original: When 1 ≤ j ≤ m ,
Revised: When 1 ≤ j ≤ m ,

Original: [Equation]
Revised: [Equation]

Original: ( x ′ i )
Revised: ( x ′ i )

Original: =
Revised: =

Original: − b i y ′ i · (cid:0) σ ′ ( ⟨ w j ,
Revised: − b i y ′ i · (cid:0) σ ′ ( ⟨ w j ,

Original: y ′ i · v ⟩ ) · y ′ i · v + σ ′ ( ⟨ w j , ξ ′ i ⟩ ) · ξ ′ i (cid:1)
Revised: y ′ i · v ⟩ ) · y ′ i · v + σ ′ ( ⟨ w j , ξ ′ i ⟩ ) · ξ ′ i (cid:1)

Original: =
Revised: =

Original: − qb i
Revised: − qb i

Original: y ′
Revised: y ′

Original: i · (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + · y ′ i · v + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1) and when m + 1 ≤ j ≤ 2 m , ∇ w j ℓ (cid:0)(cid:98)
Revised: i · (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + · y ′ i · v + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1) and when m + 1 ≤ j ≤ 2 m , ∇ w j ℓ (cid:0)(cid:98)

Original: y i · f W ( x i ) (cid:1) =
Revised: y i · f W ( x i ) (cid:1) =

Original: qc i (cid:98) y
Revised: qc i (cid:98) y

Original: i (cid:0) [ ⟨ w j , y i · v ⟩ ] q − 1 + · y i · v + [ ⟨ w j , ξ i ⟩ ] q − 1 + · ξ i (cid:1) ∇ w j ℓ (cid:0) y ′ i · f W ( x ′ i ) (cid:1)
Revised: i (cid:0) [ ⟨ w j , y i · v ⟩ ] q − 1 + · y i · v + [ ⟨ w j , ξ i ⟩ ] q − 1 + · ξ i (cid:1) ∇ w j ℓ (cid:0) y ′ i · f W ( x ′ i ) (cid:1)

Original: = qb
Revised: = qb

Original: i y
Revised: i y

Original: ′ i · (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + · y ′ i · v + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1)
Revised: ′ i · (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + · y ′ i · v + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + · ξ ′ i (cid:1)

Original: Note that ∇ w j L S ∪ S
Revised: Note that ∇ w j L S ∪ S

Original: ( W ) =
Revised: ( W ) =

Original: (cid:0) (cid:80) n u i =1 ∇ w j ℓ (cid:0)(cid:98)
Revised: (cid:0) (cid:80) n u i =1 ∇ w j ℓ (cid:0)(cid:98)

Original: y i · f W ( x i ) (cid:1)
Revised: y i · f W ( x i ) (cid:1)

Original: + (cid:80) n l i =1 ∇ w j ℓ (cid:0)
Revised: + (cid:80) n l i =1 ∇ w j ℓ (cid:0)

Original: y ′ i · f W ( x ′ i )
Revised: y ′ i · f W ( x ′ i )

Original: (cid:1)(cid:1) / ( n l + n u )+ λ · w j , we have proved the lemma.
Revised: (cid:1)(cid:1) / ( n l + n u )+ λ · w j , we have proved the lemma.

Original: A. I NNER P RODUCT U PDATE R ULE C ALCULATION
Revised: A. I NNER P RODUCT U PDATE R ULE C ALCULATION

Original: When the model is trained by gradient descent, the update rule can be formulated by
Revised: When the model is trained by gradient descent, the update rule can be formulated by

Original: [Equation]
Revised: [Equation]

Original: , j ∈ [
Revised: , j ∈ [

Original: [Equation]
Revised: [Equation]

Original: m ] .
Revised: m ] .

Original: (A.1)
Revised: (A.1)

Original: We study the performance of entire training process from two perspective: feature learning and noise memorization.
Revised: We study the performance of entire training process from two perspective: feature learning and noise memorization.

Original: Mathematically, we will focus on two quantities: ⟨ w ( t ) j , v ⟩ and ⟨ w ( t ) j , ξ l ⟩ .
Revised: Mathematically, we will focus on two quantities: ⟨ w ( t ) j , v ⟩ and ⟨ w ( t ) j , ξ l ⟩ .

Original: And then we have following lemma for the inner product update rule.
Revised: And then we have following lemma for the inner product update rule.

Original: Lemma A.2 (Inner Product Update Rule) .
Revised: Lemma A.2 (Inner Product Update Rule) .

Original: The feature learning and noise memorization performance of gradient descent can be formulated by
Revised: The feature learning and noise memorization performance of gradient descent can be formulated by

Original: [Equation]
Revised: [Equation]

Original: where j ∈ [2 m ] , l ∈ [ n u ] and u j := 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] .
Revised: where j ∈ [2 m ] , l ∈ [ n u ] and u j := 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] .

Original: Proof of Lemma A.2.
Revised: Proof of Lemma A.2.

Original: According to lemma A.1 and gradient descent update rule A.1, we have w
Revised: According to lemma A.1 and gradient descent update rule A.1, we have w

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: Taking inner product with feature vector v and noise patch ξ l and note that v is orthogonal to ξ l according to the data model
Revised: Taking inner product with feature vector v and noise patch ξ l and note that v is orthogonal to ξ l according to the data model

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: y i (cid:98)
Revised: y i (cid:98)

Original: y
Revised: y

Original: i c ( t ) i [ ⟨ w ( t ) j , y i · v ⟩ ] q − 1 + ∥ v ∥+ n l (cid:88) i =
Revised: i c ( t ) i [ ⟨ w ( t ) j , y i · v ⟩ ] q − 1 + ∥ v ∥+ n l (cid:88) i =

Original: b ( t ) i [ ⟨ w ( t ) j ,
Revised: b ( t ) i [ ⟨ w ( t ) j ,

Original: y ′ i · v ⟩ ] q − 1 + ∥ v ∥ 22 (cid:19) , ⟨ w ( t +1)
Revised: y ′ i · v ⟩ ] q − 1 + ∥ v ∥ 22 (cid:19) , ⟨ w ( t +1)

Original: j , ξ l ⟩ = (1 − ηλ ) · ⟨ w ( t ) j , ξ l ⟩ + qηu j n l + n u (cid:18) n u (cid:88) i =
Revised: j , ξ l ⟩ = (1 − ηλ ) · ⟨ w ( t ) j , ξ l ⟩ + qηu j n l + n u (cid:18) n u (cid:88) i =

Original: c ( t ) i (cid:98)
Revised: c ( t ) i (cid:98)

Original: y
Revised: y

Original: i (cid:0) [ ⟨ w j , y i · v ⟩ ] q − 1 + y i ⟨ v , ξ l ⟩ + [ ⟨ w j , ξ i ⟩ ] q − 1 + ⟨ ξ i , ξ l ⟩ (cid:1) + n l (cid:88) i =
Revised: i (cid:0) [ ⟨ w j , y i · v ⟩ ] q − 1 + y i ⟨ v , ξ l ⟩ + [ ⟨ w j , ξ i ⟩ ] q − 1 + ⟨ ξ i , ξ l ⟩ (cid:1) + n l (cid:88) i =

Original: b ( t ) i
Revised: b ( t ) i

Original: y ′
Revised: y ′

Original: i (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + y ′ i ⟨ v , ξ l ⟩ + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + ⟨ ξ ′ i , ξ l ⟩ (cid:1)(cid:19) =
Revised: i (cid:0) [ ⟨ w j , y ′ i · v ⟩ ] q − 1 + y ′ i ⟨ v , ξ l ⟩ + [ ⟨ w j , ξ ′ i ⟩ ] q − 1 + ⟨ ξ ′ i , ξ l ⟩ (cid:1)(cid:19) =

Original: (1 − ηλ ) · ⟨ w ( t ) j , ξ l ⟩ + qηu j n l + n u (cid:18) n u (cid:88) i =
Revised: (1 − ηλ ) · ⟨ w ( t ) j , ξ l ⟩ + qηu j n l + n u (cid:18) n u (cid:88) i =

Original: 1 (cid:98)
Revised: 1 (cid:98)

Original: y
Revised: y

Original: i c ( t ) i [ ⟨ w ( t ) j , ξ i ⟩ ] q − 1 + ⟨ ξ i , ξ l ⟩ + n l (cid:88) i =
Revised: i c ( t ) i [ ⟨ w ( t ) j , ξ i ⟩ ] q − 1 + ⟨ ξ i , ξ l ⟩ + n l (cid:88) i =

Original: y ′
Revised: y ′

Original: i b ( t ) i [ ⟨ w ( t ) j , ξ ′ i ⟩ ] q − 1 + ⟨ ξ ′ i , ξ l ⟩ (cid:19) , and
Revised: i b ( t ) i [ ⟨ w ( t ) j , ξ ′ i ⟩ ] q − 1 + ⟨ ξ ′ i , ξ l ⟩ (cid:19) , and

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: A. 
Revised: A.

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: ¯Λ
Revised: ¯Λ

Original: [Equation]
Revised: [Equation]

Original: − 1 = max 1 ≤ j ≤ m −⟨ w ( t ) j , v ⟩ , which characterize the feature learning aspect of training process.
Revised: − 1 = max 1 ≤ j ≤ m −⟨ w ( t ) j , v ⟩ , which characterize the feature learning aspect of training process.

Original: An easy way to distinguish between (cid:98) Λ ( t ) r and ¯Λ ( t ) r is that (cid:98) Λ ( t ) r should be large while ¯Λ ( t ) r should be small.
Revised: An easy way to distinguish between (cid:98) Λ ( t ) r and ¯Λ ( t ) r is that (cid:98) Λ ( t ) r should be large while ¯Λ ( t ) r should be small.

Original: Let Γ ( t ) i = max 1 ≤ j ≤ 2 m ⟨ w j , ξ i ⟩ , i ∈ [ n u ] , Γ ′ ( t )
Revised: Let Γ ( t ) i = max 1 ≤ j ≤ 2 m ⟨ w j , ξ i ⟩ , i ∈ [ n u ] , Γ ′ ( t )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: m ⟨ w
Revised: m ⟨ w

Original: [Equation]
Revised: [Equation]

Original: ξ ′ i ⟩ , i ∈
Revised: ξ ′ i ⟩ , i ∈

Original: [ n l ] , which characterize the noise memorization aspect of training process with respect to a particular sample.
Revised: [ n l ] , which characterize the noise memorization aspect of training process with respect to a particular sample.

Original: Let Γ (
Revised: Let Γ (

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:9) , which characterize the noise memorization aspect of training process regardless of the sample index.
Revised: (cid:9) , which characterize the noise memorization aspect of training process regardless of the sample index.

Original: We first provide the concentration inequality for (cid:98) Λ (0) r and ¯Λ (0) r in the following lemma.
Revised: We first provide the concentration inequality for (cid:98) Λ (0) r and ¯Λ (0) r in the following lemma.

Original: Lemma A.3.
Revised: Lemma A.3.

Original: With probability at least 1 − 4 δ with respect to the randomness of initialization of w ,we have
Revised: With probability at least 1 − 4 δ with respect to the randomness of initialization of w ,we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: Λ
Revised: Λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E [¯Λ
Revised: E [¯Λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.3.
Revised: Proof of lemma A.3.

Original: Note that (cid:98) Λ (0)
Revised: Note that (cid:98) Λ (0)

Original: [Equation]
Revised: [Equation]

Original: ¯Λ (0) =
Revised: ¯Λ (0) =

Original: max m +1 ≤ j ≤ 2 m ⟨ w (0) j , v ⟩ and ¯Λ (0) − 1 = max m +1 ≤ j ≤ 2 m −⟨ w (0) j , v ⟩ , w (0) j ∼ N ( 0 , σ 20 I ) and v is a fixed vector.
Revised: max m +1 ≤ j ≤ 2 m ⟨ w (0) j , v ⟩ and ¯Λ (0) − 1 = max m +1 ≤ j ≤ 2 m −⟨ w (0) j , v ⟩ , w (0) j ∼ N ( 0 , σ 20 I ) and v is a fixed vector.

Original: Therefore, ⟨ w (0) j , v ⟩ ∼ N (0 , σ 20 ∥ v ∥ 22 ) , −⟨ w (0) j , v ⟩ ∼ N (0 , σ 20 ∥ v ∥ 22 ) for all 1 ≤ j ≤ 2 m and (cid:98) Λ (0) r , ¯Λ (0) r , r ∈ {± 1 } are identically distributed.
Revised: Therefore, ⟨ w (0) j , v ⟩ ∼ N (0 , σ 20 ∥ v ∥ 22 ) , −⟨ w (0) j , v ⟩ ∼ N (0 , σ 20 ∥ v ∥ 22 ) for all 1 ≤ j ≤ 2 m and (cid:98) Λ (0) r , ¯Λ (0) r , r ∈ {± 1 } are identically distributed.

Original: Therefore, without loss of generality, we only need to discuss the concentration of (cid:98) Λ (0)1 .
Revised: Therefore, without loss of generality, we only need to discuss the concentration of (cid:98) Λ (0)1 .

Original: Then, we introduce following theorem, which is commonly used in the study of Gaussian processes, to discuss the concentration of (cid:98) Λ (0)1 , (cid:98) Λ (0) − 1 .
Revised: Then, we introduce following theorem, which is commonly used in the study of Gaussian processes, to discuss the concentration of (cid:98) Λ (0)1 , (cid:98) Λ (0) − 1 .

Original: Theorem A.4 (Borell-TIS inequality) .
Revised: Theorem A.4 (Borell-TIS inequality) .

Original: Let X be a centered Gaussian on R m and set σ 2 X := max i ∈ [ m ] E ( X 2 i ) .
Revised: Let X be a centered Gaussian on R m and set σ 2 X := max i ∈ [ m ] E ( X 2 i ) .

Original: Then for each t > 0 ,
Revised: Then for each t > 0 ,

Original: [Equation]
Revised: [Equation]

Original: By applying this theorem, we have
Revised: By applying this theorem, we have

Original: [Equation]
Revised: [Equation]

Original: For the expectation of (cid:98) Λ (0) r and ¯Λ (0) r , we give the following lemma.
Revised: For the expectation of (cid:98) Λ (0) r and ¯Λ (0) r , we give the following lemma.

Original: Lemma A.5.
Revised: Lemma A.5.

Original: Let Y = max 1 ≤ i ≤ m X i , where X i ∼ N (0 , σ 2 ) are i.i.d.
Revised: Let Y = max 1 ≤ i ≤ m X i , where X i ∼ N (0 , σ 2 ) are i.i.d.

Original: random variables.
Revised: random variables.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: By applying the lemma above, we have
Revised: By applying the lemma above, we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: Then we provide concentration inequality for Γ (0) i in the following lemma.
Revised: Then we provide concentration inequality for Γ (0) i in the following lemma.

Original: Lemma A.6.
Revised: Lemma A.6.

Original: Suppose that d ≥ Ω(log( m ( n u + n l ) /δ )) , m = Ω(log(1 /δ )) .
Revised: Suppose that d ≥ Ω(log( m ( n u + n l ) /δ )) , m = Ω(log(1 /δ )) .

Original: Then with probability at least 1 − δ ,
Revised: Then with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma A.6.
Revised: Proof of Lemma A.6.

Original: By Lemma C
Revised: By Lemma C

Original: [Equation]
Revised: [Equation]

Original: 1, with probability at least 1 − δ/ 4 ,
Revised: 1, with probability at least 1 − δ/ 4 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Gaussian tail bound and union bound, with probability at least 1 − δ/ 4 , ⟨
Revised: Therefore, by Gaussian tail bound and union bound, with probability at least 1 − δ/ 4 , ⟨

Original: [Equation]
Revised: [Equation]

Original: for i ∈ [
Revised: for i ∈ [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that P (cid:0)
Revised: Note that P (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 0) j , ξ i ⟩ (cid:1) is an absolute constant and therefore by the condition on m , we have
Revised: 0) j , ξ i ⟩ (cid:1) is an absolute constant and therefore by the condition on m , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: On the other hand, according to (A.2) and (A.3), we have
Revised: On the other hand, according to (A.2) and (A.3), we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: A. S TAGE I OF GD: O N - DIAGONAL FEATURE LEARNING
Revised: A. S TAGE I OF GD: O N - DIAGONAL FEATURE LEARNING

Original: In this stage, (cid:98) Λ ( t ) 1 and (cid:98) Λ ( t ) − 1 respectively increase to magnitude Θ(1 /m ) and ¯Λ ( t ) 1 , ¯Λ ( t ) − 1 and Γ ( t )
Revised: In this stage, (cid:98) Λ ( t ) 1 and (cid:98) Λ ( t ) − 1 respectively increase to magnitude Θ(1 /m ) and ¯Λ ( t ) 1 , ¯Λ ( t ) − 1 and Γ ( t )

Original: j remain small, the same magnitude as initialization.
Revised: j remain small, the same magnitude as initialization.

Original: In order to characterize the behaviour of feature learning and noise memorization during Stage I, we decompose the analysis into following three parts:
Revised: In order to characterize the behaviour of feature learning and noise memorization during Stage I, we decompose the analysis into following three parts:

Original: First, in lemma A.11, we provide a lower bound of the update rules of on-diagonal feature learning term of (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 to lower-bound their increasing speed, and an upper bound of off-diagonal feature learning term ¯Λ ( t ) 1 , ¯Λ ( t ) − 1 to indicate their decrease.
Revised: First, in lemma A.11, we provide a lower bound of the update rules of on-diagonal feature learning term of (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 to lower-bound their increasing speed, and an upper bound of off-diagonal feature learning term ¯Λ ( t ) 1 , ¯Λ ( t ) − 1 to indicate their decrease.

Original: Second, in lemma A.13, we provide a upper bound of the update rules of noise memorization term Γ ( t ) to upper-bound its increasing speed.
Revised: Second, in lemma A.13, we provide a upper bound of the update rules of noise memorization term Γ ( t ) to upper-bound its increasing speed.

Original: Third, we provide a useful lemma, which is a derivation of Claim C.20 in Allen-Zhu & Li (2020b), which we call tensor power method.
Revised: Third, we provide a useful lemma, which is a derivation of Claim C.20 in Allen-Zhu & Li (2020b), which we call tensor power method.

Original: By applying tensor power method, we will prove that:
Revised: By applying tensor power method, we will prove that:

Original: • When (cid:98) Λ ( t ) 1 reaches Θ(1 /m ) at T 1 , ¯Λ ( t ) 1 and Γ ( t ) remain a magnitude no more than initialization.
Revised: • When (cid:98) Λ ( t ) 1 reaches Θ(1 /m ) at T 1 , ¯Λ ( t ) 1 and Γ ( t ) remain a magnitude no more than initialization.

Original: • When (cid:98) Λ ( t ) − 1 reaches Θ(1 /m ) at T − 1 , ¯Λ − 1 and Γ ( t ) remain a magnitude no more than initialization.
Revised: • When (cid:98) Λ ( t ) − 1 reaches Θ(1 /m ) at T − 1 , ¯Λ − 1 and Γ ( t ) remain a magnitude no more than initialization.

Original: A.4. U PPER BOUND AND LOWER BOUND FOR (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 AND ¯Λ ( t ) 1 , ¯Λ ( t ) −
Revised: A.4. U PPER BOUND AND LOWER BOUND FOR (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 AND ¯Λ ( t ) 1 , ¯Λ ( t ) −

Original: We first consider Stage I of GD when max r ∈{ +1 , − 1 } (cid:8)(cid:98) Λ ( t ) r , ¯Λ ( t ) r (cid:9) ≤ Θ( m − 1 ) .
Revised: We first consider Stage I of GD when max r ∈{ +1 , − 1 } (cid:8)(cid:98) Λ ( t ) r , ¯Λ ( t ) r (cid:9) ≤ Θ( m − 1 ) .

Original: In this stage, we first prove following lemma:
Revised: In this stage, we first prove following lemma:

Original: Lemma A.7.
Revised: Lemma A.7.

Original: As long as max r ∈{ +1 , − 1 } (cid:8)(cid:98) Λ ( t ) r , ¯Λ ( t ) r (cid:9) ≤ Θ( m − 1 ) , we have c ( t ) i := − ℓ ′ (cid:0)(cid:98)
Revised: As long as max r ∈{ +1 , − 1 } (cid:8)(cid:98) Λ ( t ) r , ¯Λ ( t ) r (cid:9) ≤ Θ( m − 1 ) , we have c ( t ) i := − ℓ ′ (cid:0)(cid:98)

Original: y i · f W ( t ) ( x i ) (cid:1) and b ( t ) i := − ℓ ′ (cid:0)
Revised: y i · f W ( t ) ( x i ) (cid:1) and b ( t ) i := − ℓ ′ (cid:0)

Original: y ′ i · f W ( t ) ( x ′ i ) (cid:1) remains 1 / 2 ±
Revised: y ′ i · f W ( t ) ( x ′ i ) (cid:1) remains 1 / 2 ±

Original: o (1) .
Revised: o (1) .

Original: Proof of lemma A.7.
Revised: Proof of lemma A.7.

Original: Note that ℓ ( z )
Revised: Note that ℓ ( z )

Original: =
Revised: =

Original: log(1 + exp ( − z )) and − ℓ ′ ( z ) =
Revised: log(1 + exp ( − z )) and − ℓ ′ ( z ) =

Original: exp ( − z ) / (cid:0) 1 + exp ( − z ) (cid:1) , and without loss of generality assuming (cid:98) y i =
Revised: exp ( − z ) / (cid:0) 1 + exp ( − z ) (cid:1) , and without loss of generality assuming (cid:98) y i =

Original: y i
Revised: y i

Original: = 1 , we can express c ( t ) i as follow:
Revised: = 1 , we can express c ( t ) i as follow:

Original: [Equation]
Revised: [Equation]

Original: Since σ ( ⟨ w ( t ) j , v ⟩ ) dominates σ ( ⟨ w ( t ) j , ξ ⟩ ) for j ∈ [ m ] , which will be proved later by using tensor power method , we have
Revised: Since σ ( ⟨ w ( t ) j , v ⟩ ) dominates σ ( ⟨ w ( t ) j , ξ ⟩ ) for j ∈ [ m ] , which will be proved later by using tensor power method , we have

Original: [Equation]
Revised: [Equation]

Original: On the one side,
Revised: On the one side,

Original: [Equation]
Revised: [Equation]

Original: On the other side, according to lemma 5.3, we have ¯Λ ( t )=
Revised: On the other side, according to lemma 5.3, we have ¯Λ ( t )=

Original: (cid:101) O
Revised: (cid:101) O

Original: ( d − 14 ) and Γ ( t ) = (cid:101)
Revised: ( d − 14 ) and Γ ( t ) = (cid:101)

Original: O
Revised: O

Original: ( d − 14 + ϵ ) , it follows that
Revised: ( d − 14 + ϵ ) , it follows that

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have c ( t ) i
Revised: Therefore, we have c ( t ) i

Original: = 1 / 2 ± o (1) if (cid:98)
Revised: = 1 / 2 ± o (1) if (cid:98)

Original: y i
Revised: y i

Original: =
Revised: =

Original: y i
Revised: y i

Original: = 1 and other cases ( (cid:98)
Revised: = 1 and other cases ( (cid:98)

Original: y i
Revised: y i

Original: =
Revised: =

Original: y i
Revised: y i

Original: = 1 , (cid:98) y
Revised: = 1 , (cid:98) y

Original: i = − y i , b ( t ) i ) can be proved in a similar way.
Revised: i = − y i , b ( t ) i ) can be proved in a similar way.

Original: By applying above lemma, we can obtain following lemma:
Revised: By applying above lemma, we can obtain following lemma:

Original: Lemma A.8.
Revised: Lemma A.8.

Original: For any δ < 1 / 2 , with probability at least 1 − 2 δ over pseudolabels generated by the pseudolabeler, we have
Revised: For any δ < 1 / 2 , with probability at least 1 − 2 δ over pseudolabels generated by the pseudolabeler, we have

Original: [Equation]
Revised: [Equation]

Original: where o (1) is with respect to d .
Revised: where o (1) is with respect to d .

Original: If we denote { ( x i , y i ) | y i
Revised: If we denote { ( x i , y i ) | y i

Original: = 1 , i ∈
Revised: = 1 , i ∈

Original: [ n u ] } as S 1 , { ( x i , y i ) |
Revised: [ n u ] } as S 1 , { ( x i , y i ) |

Original: y i = − 1 , i ∈
Revised: y i = − 1 , i ∈

Original: [ n u ] } as S − 1 , | S 1 | as n 1 and | S − 1 | as n − 1 , we have with probability at least 1 − 4 δ that
Revised: [ n u ] } as S − 1 , | S 1 | as n 1 and | S − 1 | as n − 1 , we have with probability at least 1 − 4 δ that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.8.
Revised: Proof of lemma A.8.

Original: First, according to lemma A.7, we have
Revised: First, according to lemma A.7, we have

Original: [Equation]
Revised: [Equation]

Original: Then, according to Hoeffding’s inequality when a i = − 1 , b i = 1 , we have
Revised: Then, according to Hoeffding’s inequality when a i = − 1 , b i = 1 , we have

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Note that the pseudolabel (cid:98) y i generated by the pseudolabeler takes y i with probability p and − y
Revised: Note that the pseudolabel (cid:98) y i generated by the pseudolabeler takes y i with probability p and − y

Original: i with probability 1 − p , we have E (cid:2) 1 n u (cid:80) n u
Revised: i with probability 1 − p , we have E (cid:2) 1 n u (cid:80) n u

Original: i
Revised: i

Original: =
Revised: =

Original: 1 (cid:98)
Revised: 1 (cid:98)

Original: y
Revised: y

Original: i y i (cid:3) = 1 n u (cid:80) n u i =1 E (cid:2)(cid:98)
Revised: i y i (cid:3) = 1 n u (cid:80) n u i =1 E (cid:2)(cid:98)

Original: y
Revised: y

Original: i y i (cid:3) = 2 p − 1 .
Revised: i y i (cid:3) = 2 p − 1 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: A.5) holds with probability at least 1 − 2 δ .
Revised: A.5) holds with probability at least 1 − 2 δ .

Original: According to (A.4) and (A.5), we have
Revised: According to (A.4) and (A.5), we have

Original: [Equation]
Revised: [Equation]

Original: which verifies the first statement of the lemma
Revised: which verifies the first statement of the lemma

Original: And the other part of the lemma can be proved in a similar way
Revised: And the other part of the lemma can be proved in a similar way

Original: [Equation]
Revised: [Equation]

Original: According to above lemma and note that n u , n 1 , n − 1 = ω (1)
Revised: According to above lemma and note that n u , n 1 , n − 1 = ω (1)

Original: [Equation]
Revised: [Equation]

Original: we have further that
Revised: we have further that

Original: [Equation]
Revised: [Equation]

Original: n u
Revised: n u

Original: [Equation]
Revised: [Equation]

Original: with high probability.
Revised: with high probability.

Original: Besides, we also need an approximation about n 1 and n − 1 , which is given as the following lemma:
Revised: Besides, we also need an approximation about n 1 and n − 1 , which is given as the following lemma:

Original: Lemma A.9.
Revised: Lemma A.9.

Original: For r ∈ {± 1 } , it holds with probability at least 1 − 2 δ that
Revised: For r ∈ {± 1 } , it holds with probability at least 1 − 2 δ that

Original: [Equation]
Revised: [Equation]

Original: where n r := |{ ( x i , y i ) | y i
Revised: where n r := |{ ( x i , y i ) | y i

Original: =
Revised: =

Original: r, i ∈ [ n u ] }| .
Revised: r, i ∈ [ n u ] }| .

Original: Proof of lemma A.9.
Revised: Proof of lemma A.9.

Original: Note that n r = (cid:80) n u i =1 

Revised: Note that n r = (cid:80) n u i =1

Original: [ X i = r ] , r ∈ {± 1 } where X i takes label +1 or −with equal probability 1 / 2 , according to Hoeffding’s inequality, we have
Revised: [ X i = r ] , r ∈ {± 1 } where X i takes label +1 or −with equal probability 1 / 2 , according to Hoeffding’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: and it follows that
Revised: and it follows that

Original: [Equation]
Revised: [Equation]

Original: leading to
Revised: leading to

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ .
Revised: with probability at least 1 − 2 δ .

Original: For labeled dataset S ′ =
Revised: For labeled dataset S ′ =

Original: { ( x ′ i , y ′ i ) } n l i =1 , we also have
Revised: { ( x ′ i , y ′ i ) } n l i =1 , we also have

Original: Lemma A.10.
Revised: Lemma A.10.

Original: For r ∈ {± 1 } , it holds with probability at least 1 − 2 δ that
Revised: For r ∈ {± 1 } , it holds with probability at least 1 − 2 δ that

Original: [Equation]
Revised: [Equation]

Original: where n ′ r := |{ ( x ′ i , y ′ i ) | y ′ i =
Revised: where n ′ r := |{ ( x ′ i , y ′ i ) | y ′ i =

Original: r, i ∈ [ n l ] }| .
Revised: r, i ∈ [ n l ] }| .

Original: Then we are prepared to estimate an lower bound of increasing speed of (cid:98) Λ ( t ) and an upper bound of decreasing speed of ¯Λ ( t ) in the following lemma.
Revised: Then we are prepared to estimate an lower bound of increasing speed of (cid:98) Λ ( t ) and an upper bound of decreasing speed of ¯Λ ( t ) in the following lemma.

Original: Lemma A.11.
Revised: Lemma A.11.

Original: For (cid:98) Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and (cid:98) Λ ( t ) − 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ , we have with high probability that
Revised: For (cid:98) Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and (cid:98) Λ ( t ) − 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ , we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: For ¯Λ ( t ) 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and ¯Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , − v ⟩ , we have with high probability that
Revised: For ¯Λ ( t ) 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and ¯Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , − v ⟩ , we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.11.
Revised: Proof of lemma A.11.

Original: We first prove the former inequality.
Revised: We first prove the former inequality.

Original: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = 1 , then we have (cid:98) Λ ( t +1)≥ ⟨ w ( t +1)
Revised: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = 1 , then we have (cid:98) Λ ( t +1)≥ ⟨ w ( t +1)

Original: j ∗ , v ⟩
Revised: j ∗ , v ⟩

Original: = (1 − ηλ ) · ⟨ w ( t ) j ∗ , v ⟩ + 3 η n l + n u (cid:18) n u (cid:88)
Revised: = (1 − ηλ ) · ⟨ w ( t ) j ∗ , v ⟩ + qη n l + n u (cid:18) n u (cid:88)

Original: [Equation]
Revised: [Equation]

Original: = y i (cid:98) y i c ( t ) i [ ⟨ w ( t ) j ∗ , y i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125) ♣ + n l (cid:88) i =
Revised: = y i (cid:98) y i c ( t ) i [ ⟨ w ( t ) j ∗ , y i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125) ♣ + n l (cid:88) i =

Original: b
Revised: b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then we respectively estimate terms ♣ and ⋆ .
Revised: Then we respectively estimate terms ♣ and ⋆ .

Original: For ♣ , note the definition of j ∗ that (cid:98) Λ ( t ) 1 = ⟨ w ( t ) j ∗ , v ⟩ and note the increasing property of (cid:98) Λ ( t ) 1 and (cid:98) Λ (0)1 > 0 with high probability, we have ⟨ w ( t ) j ∗ , v ⟩ > 0 .
Revised: For ♣ , note the definition of j ∗ that (cid:98) Λ ( t ) 1 = ⟨ w ( t ) j ∗ , v ⟩ and note the increasing property of (cid:98) Λ ( t ) 1 and (cid:98) Λ (0)1 > 0 with high probability, we have ⟨ w ( t ) j ∗ , v ⟩ > 0 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: where S 1 := { ( x i , y i )
Revised: where S 1 := { ( x i , y i )

Original: | y
Revised: | y

Original: i = 1 , i ∈ [ n u ] } , S − 1 :
Revised: i = 1 , i ∈ [ n u ] } , S − 1 :

Original: = { ( x i , y i )
Revised: = { ( x i , y i )

Original: | y
Revised: | y

Original: i = − 1 , i ∈
Revised: i = − 1 , i ∈

Original: [ n u ] } , n 1 = | S 1 | and the last equality is due to (A.6).
Revised: [ n u ] } , n 1 = | S 1 | and the last equality is due to (A.6).

Original: For ⋆ , similarly we have
Revised: For ⋆ , similarly we have

Original: [Equation]
Revised: [Equation]

Original: where S ′ 

Revised: where S ′

Original: = { ( x ′ i , y ′ i )
Revised: = { ( x ′ i , y ′ i )

Original: |
Revised: |

Original: y ′ i = 1 , i ∈ [ n l ] } , S ′ − 1 = { ( x ′ i , y ′ i )
Revised: y ′ i = 1 , i ∈ [ n l ] } , S ′ − 1 = { ( x ′ i , y ′ i )

Original: |
Revised: |

Original: y ′
Revised: y ′

Original: i = − 1 , i ∈
Revised: i = − 1 , i ∈

Original: [ n l ] } , n ′ 1 =
Revised: [ n l ] } , n ′ 1 =

Original: | S ′ 1 | and the last equality is due to lemma A.7.
Revised: | S ′ 1 | and the last equality is due to lemma A.7.

Original: According to (A.7) and (A.8), we have (cid:98) Λ (
Revised: According to (A.7) and (A.8), we have (cid:98) Λ (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:19) · ∥ v ∥ 22 · (cid:0)(cid:98) Λ ( t ) 1 (cid:1) q − 1 .
Revised: · ∥ v ∥ 22 · (cid:0)(cid:98) Λ ( t ) 1 (cid:1) q − 1 .

Original: (A.9)
Revised: (A.9)

Original: According to lemma A.9 and lemma A.10, and note that n l = (cid:101) Θ(1) , n u = ω ( d 4 ϵ ) , we have for ♠ that with probability at least 1 − 4 δ(cid:12)(cid:12)(cid:12)(cid:12) n [Equation] (cid:17)
Revised: According to lemma A.9 and lemma A.10, and note that n l = (cid:101) Θ(1) , n u = ω ( d 4 ϵ ) , we have for ♠ that with probability at least 1 − 4 δ

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: =

Original: Θ [Equation] u (cid:17) = o (1)
Revised: = o (1)

Original: Therefore, note that n u = ω ( n l ) and n u = ω (1) , we have
Revised: Therefore, note that n u = ω ( n l ) and n u = ω (1) , we have

Original: [Equation]
Revised: [Equation]

Original: Plugging (A.10) into (A.9), we have
Revised: Plugging (A.10) into (A.9), we have

Original: [Equation]
Revised: [Equation]

Original: which verifies the first inequality of case r = 1 in the lemma.
Revised: which verifies the first inequality of case r = 1 in the lemma.

Original: Let j ∗∗ = argmax m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ and note that u j ∗∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = − 1 , we have
Revised: Let j ∗∗ = argmax m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ and note that u j ∗∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: For ♣ , note the definition of j ∗∗ that (cid:98) Λ ( t ) − 1 = ⟨ w ( t ) j ∗∗ , − v ⟩ and note the increasing property of (cid:98) Λ ( t ) −and (cid:98) Λ (0) − 1 > 0 with high probability, we have ⟨ w ( t ) j ∗∗ , − v ⟩ > 0 .
Revised: For ♣ , note the definition of j ∗∗ that (cid:98) Λ ( t ) − 1 = ⟨ w ( t ) j ∗∗ , − v ⟩ and note the increasing property of (cid:98) Λ ( t ) −and (cid:98) Λ (0) − 1 > 0 with high probability, we have ⟨ w ( t ) j ∗∗ , − v ⟩ > 0 .

Original: According to (A.6), it follows that n
Revised: According to (A.6), it follows that n

Original: [Equation]
Revised: [Equation]

Original: where S − 1 :
Revised: where S − 1 :

Original: = { ( x i , y i )
Revised: = { ( x i , y i )

Original: | y
Revised: | y

Original: i = − 1 , i ∈
Revised: i = − 1 , i ∈

Original: [ n u ] } , n − 1 = | S − 1 | .
Revised: [ n u ] } , n − 1 = | S − 1 | .

Original: For ⋆ , according to lemma A.7, similarly we have n l
Revised: For ⋆ , according to lemma A.7, similarly we have n l

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: where S ′− 1 = { ( x ′ i , y ′ i )
Revised: where S ′− 1 = { ( x ′ i , y ′ i )

Original: |
Revised: |

Original: y ′
Revised: y ′

Original: i = − 1 , i ∈
Revised: i = − 1 , i ∈

Original: [ n l ] } and n ′− 1 = | S ′− 1 | .
Revised: [ n l ] } and n ′− 1 = | S ′− 1 | .

Original: According to (A.12) and (A.13), we have
Revised: According to (A.12) and (A.13), we have

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: (A.

Original: [Equation]
Revised: [Equation]

Original: According to lemma A.9 and lemma A.10, and note that n l = (cid:101) Θ(1) , n u = ω ( d 4 ϵ ) , we have for ♠ that with probability at least 1 − 4 δ
Revised: According to lemma A.9 and lemma A.10, and note that n l = (cid:101) Θ(1) , n u = ω ( d 4 ϵ ) , we have for ♠ that with probability at least 1 − 4 δ

Original: [Equation]
Revised: [Equation]

Original: Therefore, note that n u = ω ( n l ) and n u = ω (1) , we have n −
Revised: Therefore, note that n u = ω ( n l ) and n u = ω (1) , we have n −

Original: [Equation]
Revised: [Equation]

Original: Plugging (A.15) into (A.14), we have
Revised: Plugging (A.15) into (A.14), we have

Original: [Equation]
Revised: [Equation]

Original: which verifies the first inequality of case r = − 1 in the lemma.
Revised: which verifies the first inequality of case r = − 1 in the lemma.

Original: Next, we prove the latter part of the lemma.
Revised: Next, we prove the latter part of the lemma.

Original: Let j ♮ =
Revised: Let j ♮ =

Original: arg max m +1 ≤ j ≤ 2 m ⟨ w ( t +1)
Revised: arg max m +1 ≤ j ≤ 2 m ⟨ w ( t +1)

Original: j , v ⟩ , then we have:
Revised: j , v ⟩ , then we have:

Original: [Equation]
Revised: [Equation]

Original: For ♣ , according to (A.6), we have n
Revised: For ♣ , according to (A.6), we have n

Original: [Equation]
Revised: [Equation]

Original: and for ⋆ it’s obvious that
Revised: and for ⋆ it’s obvious that

Original: [Equation]
Revised: [Equation]

Original: Therefore, it follows that
Revised: Therefore, it follows that

Original: [Equation]
Revised: [Equation]

Original: Let j ♮♮
Revised: Let j ♮♮

Original: =
Revised: =

Original: arg max
Revised: arg max

Original: [Equation]
Revised: [Equation]

Original: , − v ⟩ , then we have:
Revised: , − v ⟩ , then we have:

Original: [Equation]
Revised: [Equation]

Original: which verifies the second part of the lemma.
Revised: which verifies the second part of the lemma.

Original: Although the accuracy of pseudolabeler is larger than 1 / 2 , which is used as an assumption in the previous proof, we can also analyse the model with high label flipping probability and the accuracy of pseudolabeler p is smaller than 1 / 2 .
Revised: Although the accuracy of pseudolabeler is larger than 1 / 2 , which is used as an assumption in the previous proof, we can also analyse the model with high label flipping probability and the accuracy of pseudolabeler p is smaller than 1 / 2 .

Original: In this case, the neural network for pretraining will turn to fit the opposite direction of feature vector, ¯Λ ( t ) r will increase and (cid:98) Λ ( t ) r will decrease, which is formulated as the following lemma.
Revised: In this case, the neural network for pretraining will turn to fit the opposite direction of feature vector, ¯Λ ( t ) r will increase and (cid:98) Λ ( t ) r will decrease, which is formulated as the following lemma.

Original: Lemma A.12.
Revised: Lemma A.12.

Original: For (cid:98) Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and (cid:98) Λ ( t ) − 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ , we have with high probability that (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t ) r , r ∈ {± 1 } .
Revised: For (cid:98) Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , v ⟩ and (cid:98) Λ ( t ) − 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , − v ⟩ , we have with high probability that (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t ) r , r ∈ {± 1 } .

Original: For ¯Λ ( t ) 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and ¯Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , − v ⟩ , we have with high probability that
Revised: For ¯Λ ( t ) 1 := max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and ¯Λ ( t ) 1 := max 1 ≤ j ≤ m ⟨ w ( t ) j , − v ⟩ , we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.12.
Revised: Proof of lemma A.12.

Original: First, we prove the former part of this lemma.
Revised: First, we prove the former part of this lemma.

Original: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)
Revised: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)

Original: j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = 1 , then we have (cid:98) Λ ( t +1)= ⟨ w ( t +1)
Revised: j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] = 1 , then we have (cid:98) Λ ( t +1)= ⟨ w ( t +1)

Original: j ∗ , v ⟩
Revised: j ∗ , v ⟩

Original: = (1 − ηλ ) · ⟨ w ( t ) j ∗ , v ⟩ + 3 η n l + n u (cid:18) n u (cid:88) i =
Revised: = (1 − ηλ ) · ⟨ w ( t ) j ∗ , v ⟩ + qη n l + n u (cid:18) n u (cid:88) i =

Original: y i (cid:98)
Revised: y i (cid:98)

Original: y
Revised: y

Original: i c ( t ) i [ ⟨ w ( t ) j ∗ , y i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125) ♣ + n l (cid:88) i =
Revised: i c ( t ) i [ ⟨ w ( t ) j ∗ , y i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125) ♣ + n l (cid:88) i =

Original: b ( t ) i [ ⟨ w ( t ) j ∗ ,
Revised: b ( t ) i [ ⟨ w ( t ) j ∗ ,

Original: y ′ i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125)
Revised: y ′ i · v ⟩ ] q − 1 + ∥ v ∥(cid:124) (cid:123)(cid:122) (cid:125)

Original: ⋆ (cid:19) .
Revised: ⋆ (cid:19) .

Original: For ♣ , according to (A.6), we have
Revised: For ♣ , according to (A.6), we have

Original: [Equation]
Revised: [Equation]

Original: For ⋆ , according to (A.6), we have
Revised: For ⋆ , according to (A.6), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: According to lemma A.9 and note that n u = ω ( n l )
Revised: According to lemma A.9 and note that n u = ω ( n l )

Original: [Equation]
Revised: [Equation]

Original: it holds with probability at least 1 − 8 δ that n ′ 1 · (cid:16) 1 2 ± o (1) (cid:17) ≤ (cid:18) n
Revised: it holds with probability at least 1 − 8 δ that n ′ 1 · (cid:16) 1 2 ± o (1) (cid:17) ≤ (cid:18) n

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n ′− 1 · (cid:16)
Revised: n ′− 1 · (cid:16)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:17) , leading to ♣ + ⋆ ≤ 0 .
Revised: 1) (cid:17) , leading to ♣ + ⋆ ≤ 0 .

Original: Therefore, (cid:98) Λ ( t +1)≤ (1 − ηλ ) ⟨ w ( t ) j ∗ , v ⟩ ≤ (1 − ηλ ) · (cid:98) Λ ( t ) 1 .
Revised: Therefore, (cid:98) Λ ( t +1)≤ (1 − ηλ ) ⟨ w ( t ) j ∗ , v ⟩ ≤ (1 − ηλ ) · (cid:98) Λ ( t ) 1 .

Original: And we can prove in a similar way that (cid:98) Λ ( t +1)
Revised: And we can prove in a similar way that (cid:98) Λ ( t +1)

Original: −≤ (1 − ηλ ) · (cid:98) Λ ( t ) − 1 .
Revised: −≤ (1 − ηλ ) · (cid:98) Λ ( t ) − 1 .

Original: Next, we prove the second part of the lemma.
Revised: Next, we prove the second part of the lemma.

Original: Let j ♮ =
Revised: Let j ♮ =

Original: arg max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and note that u j ♮ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ]
Revised: arg max m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , v ⟩ and note that u j ♮ = 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ]

Original: = − 1 , then we have
Revised: = − 1 , then we have

Original: [Equation]
Revised: [Equation]

Original: For ♣ , note the definition of j ♮ that ¯Λ ( t ) 1 = ⟨ w ( t ) j ♮ , v ⟩ and note the increasing property of ¯Λ ( t ) 1 in this case and ¯Λ (0)1 > 0 with high probability, we have ⟨ w ( t ) j ♮ , v ⟩ > 0 .
Revised: For ♣ , note the definition of j ♮ that ¯Λ ( t ) 1 = ⟨ w ( t ) j ♮ , v ⟩ and note the increasing property of ¯Λ ( t ) 1 in this case and ¯Λ (0)1 > 0 with high probability, we have ⟨ w ( t ) j ♮ , v ⟩ > 0 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: For ⋆ , similarly we have n
Revised: For ⋆ , similarly we have n

Original: [Equation]
Revised: [Equation]

Original: According to lemma A.9, (A.17) and (A.18)
Revised: According to lemma A.9, (A.17) and (A.18)

Original: [Equation]
Revised: [Equation]

Original: we have n ′ 

Revised: we have n ′

Original: = o ( n 1 ) with high probability, therefore
Revised: = o ( n 1 ) with high probability, therefore

Original: [Equation]
Revised: [Equation]

Original: leading to
Revised: leading to

Original: [Equation]
Revised: [Equation]

Original: And we can prove in a similar way that
Revised: And we can prove in a similar way that

Original: [Equation]
Revised: [Equation]

Original: In this case ( p < 1 / 2) , given a small amount of labeled data, downstream task parameter a will learn the negative direction and the main theorems still hold
Revised: In this case ( p < 1 / 2) , given a small amount of labeled data, downstream task parameter a will learn the negative direction and the main theorems still hold

Original: [Equation]
Revised: [Equation]

Original: A.4. U NIFORM UPPER BOUND FOR Γ ( t )
Revised: A.4. U NIFORM UPPER BOUND FOR Γ ( t )

Original: The following lemma provides an upper bound for the increasing rate of Γ ( t ) .
Revised: The following lemma provides an upper bound for the increasing rate of Γ ( t ) .

Original: Lemma A.13.
Revised: Lemma A.13.

Original: For Γ ( t ) i
Revised: For Γ ( t ) i

Original: [Equation]
Revised: [Equation]

Original: Γ ( t ) :=
Revised: Γ ( t ) :=

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: ] Γ ( t )
Revised: ] Γ ( t )

Original: [Equation]
Revised: [Equation]

Original: max i
Revised: max i

Original: [Equation]
Revised: [Equation]

Original: [ n l ] Γ ′ ( t ) i } , we have with high probability that
Revised: [ n l ] Γ ′ ( t ) i } , we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: where ϵ = 0 .
Revised: where ϵ = 0 .

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.13.
Revised: Proof of lemma A.13.

Original: We first prove the former inequality.
Revised: We first prove the former inequality.

Original: Let j ⋆ = arg max 1 ≤ j ≤ 2 m ⟨ w ( t +1)
Revised: Let j ⋆ = arg max 1 ≤ j ≤ 2 m ⟨ w ( t +1)

Original: j , ξ l ⟩
Revised: j , ξ l ⟩

Original: [Equation]
Revised: [Equation]

Original: where l ∈ [ n u ] is fixed.
Revised: where l ∈ [ n u ] is fixed.

Original: According to lemma A.2, we have
Revised: According to lemma A.2, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to triangle inequality.
Revised: where the last inequality is due to triangle inequality

Revised: [Equation]

Original: For ♣ , note that l ∈ [ n u ] and there exists an i ∈ [ n u ] equivalent to l , it follows that 
Revised: For ♣ , note that l ∈ [ n u ] and there exists an i ∈ [ n u ] equivalent to l , it follows that [Equation] j ⋆ [Equation]

Revised: i ⟩ ]

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Original:  where the inequality is due to lemma A.7, ∥ ξ l ∥ 22 = (cid:101)
Revised: c [Equation] [Equation] A.20) where the inequality is due to lemma A.7, ∥ ξ l ∥ 22 = (cid:101)

Original: Θ
Revised: Θ

Original: [Equation]
Revised: [Equation]

Original: dσ 2 p)
Revised: dσ [Equation] p

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: |

Original: [Equation]
Revised: [Equation]

Original: Θ
Revised: Θ

Original: [Equation]
Revised: [Equation]

Original: d 
Revised: d 12 σ

Original: [Equation]
Revised: [Equation]

Original:  ) = (cid:101) Θ( d 12 +2 ϵ ) according to Lemma C.1 and the definition of Γ ( t ) .
Revised: p ) = (cid:101) Θ( d 12 +2 ϵ ) according to Lemma C.1 and the definition of Γ ( t ) .

Original: For ⋆ , we have n l (cid:88)
Revised: For ⋆ , we have n l [Equation]

Original: i 
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: | [Equation] −

Original: [Equation]
Revised: [Equation]

Original:  Plugging (A.20) and (A.21) into (A.19), we have Γ ( t +1) l
Revised: (A.21) Plugging (A.20) and (A.21) into (A.19), we have Γ ( t +1) l

Original: [Equation]
Revised: [Equation]

Original: which is the first part of this lemma.
Revised: which is the first part of this lemma.

Original: Let j
Revised: Let j

Original: [Equation]
Revised: [Equation]

Original: = argmax
Revised: = argmax

Original: [Equation]
Revised: [Equation]

Original: j , ξ ′ l ⟩ , where l ∈ [ n l ] is fixed.
Revised: , ξ ′ l ⟩ , where l ∈ [ n l ] is fixed.

Original: According to lemma A.2, we have
Revised: According to lemma A.2, we have

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: [Equation]

Original: w

Original: [Equation]

Original: t

Original: [Equation]

Original: j

Original: [Equation]

Original: For ♣ , we have
Revised: For ♣ , we have

Original: [Equation]
Revised: [Equation]

Original: where the inequality is due to lemma A.7, |⟨ ξ i , ξ l ⟩| = (cid:101)
Revised: where the inequality is due to lemma A.7, |⟨ ξ i , ξ l ⟩| = (cid:101)

Original: Θ( d 12 σ 2 p )
Revised: Θ( d 12 σ 2 p )

Original: = (cid:101) Θ( d 12 +2 ϵ ) and the definition of Γ ( t ) .
Revised: = (cid:101) Θ( d 12 +2 ϵ ) and the definition of Γ ( t ) .

Original: For ⋆
Revised: For ⋆

Original: [Equation]
Revised: [Equation]

Original: note that l ∈ [ n l ] and there exists an i ∈
Revised: note that l ∈ [ n l ] and there exists an i ∈

Original: [ n l ] equivalent to l
Revised: [ n l ] equivalent to l

Original: [Equation]
Revised: [Equation]

Original: it follows that
Revised: it follows that

Original: [Equation]
Revised: [Equation]

Original: (A.24)
Revised: A.24)

Original: Plugging (A.23) and (A.24) into (A.22), we have Γ ′
Revised: Plugging (A.23) and (A.24) into (A.22), we have Γ ′

Original: ( t +1) l
Revised: ( t +1) l

Original: [Equation]
Revised: [Equation]

Original: which verifies the second inequality in this lemma.
Revised: which verifies the second inequality in this lemma.

Original: Note that Γ ( t ) = max { max l ∈ [ n u ] Γ ( t ) l , max l ∈ [ n l ] Γ ′ ( t ) l } , without loss of generality, we assume Γ ( t ) =
Revised: Note that Γ ( t ) = max { max l ∈ [ n u ] Γ ( t ) l , max l ∈ [ n l ] Γ ′ ( t ) l } , without loss of generality, we assume Γ ( t ) =

Original: max l ∈ [ n u ] Γ ( t ) l and assume l ∗ =
Revised: max l ∈ [ n u ] Γ ( t ) l and assume l ∗ =

Original: argmax l ∈ [ n u ] Γ ( t +1) l , we have
Revised: argmax l ∈ [ n u ] Γ ( t +1) l , we have

Original: [Equation]
Revised: [Equation]

Original: ( t +1) =
Revised: ( t +1) =

Revised: [Equation]

Original: Γ ( t +1) l ∗
Revised: ( t +1) l ∗

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: Γ ( t ) (cid:1) q − 1 , which verifies the third inequality in this lemma.
Revised: ( t ) (cid:1) q − 1 , which verifies the third inequality in this lemma.

Original: A.4. P ROVE OFF - DIAGONAL CORRELATION REMAINS INITIALIZATION MAGNITUDE DURING [0 , T r ]
Revised: A.4. P ROVE OFF - DIAGONAL CORRELATION REMAINS INITIALIZATION MAGNITUDE DURING [0 , T r ]

Original: If the accuracy of pseudolabeler p > 1 / 2 , we have off-diagonal correlation ¯Λ ( t +1) r ≤ (1 − ηλ ) · ¯Λ ( t ) r for r ∈ { +1 , − 1 } , therefore, ¯Λ ( t ) r
Revised: If the accuracy of pseudolabeler p > 1 / 2 , we have off-diagonal correlation ¯Λ ( t +1) r ≤ (1 − ηλ ) · ¯Λ ( t ) r for r ∈ { +1 , − 1 } , therefore, ¯Λ ( t ) r

Original: = O (¯Λ (0) r )
Revised: = O (¯Λ (0) r )

Original: = (cid:101) O
Revised: = (cid:101) O

Original: ( d − 14 ) .
Revised: ( d − 14 ) .

Original: If p < 1 / 2 , we have off-diagonal correlation (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t ) r for r ∈ { +1 , − 1 } , therefore, (cid:98) Λ ( t ) r
Revised: If p < 1 / 2 , we have off-diagonal correlation (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t ) r for r ∈ { +1 , − 1 } , therefore, (cid:98) Λ ( t ) r

Original: = O ( (cid:98) Λ (0) r ) =
Revised: = O ( (cid:98) Λ (0) r ) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: ( d − 14 ) .
Revised: ( d − 14 ) .

Original: A.4. T ENSOR P OWER M ETHOD : P ROVING Γ ( t ) = O (Γ (0) ) DURING [0 , T r ] AND COMPUTING THE MAGNITUDE OF T r
Revised: A.4. T ENSOR P OWER M ETHOD : P ROVING Γ ( t ) = O (Γ (0) ) DURING [0 , T r ] AND COMPUTING THE MAGNITUDE OF T r

Original: According to section A.4.1 and A.4.2, we have obtained following upper bounds and lower bounds for feature learning term (cid:98) Λ ( t ) r , ¯Λ ( t ) r , r ∈ { +1 , − 1 } and noise memorization term Γ ( t ) :
Revised: According to section A.4.1 and A.4.2, we have obtained following upper bounds and lower bounds for feature learning term (cid:98) Λ ( t ) r , ¯Λ ( t ) r , r ∈ { +1 , − 1 } and noise memorization term Γ ( t ) :

Original: When t ∈ [0 , T r ] , we have (cid:98) Λ ( t +1) r ≥ (cid:98)
Revised: When t ∈ [0 , T r ] , we have (cid:98) Λ ( t +1) r ≥ (cid:98)

Original: [Equation]
Revised: [Equation]

Original: ( t ) r +
Revised: ( t ) r +

Original: [Equation]
Revised: [Equation]

Original: (2 p
Revised: (2 p

Original: [Equation]
Revised: [Equation]

Original: 1) · Θ( d ) · ( (cid:98) Λ ( t ) r ) q − 1 and ¯Λ ( t +1) r ≤ (1 − ηλ ) · ¯Λ ( t ) r , for r ∈ { +1 , − 1 } ;
Revised: 1) · Θ( d ) · ( (cid:98) Λ ( t ) r ) q − 1 and ¯Λ ( t +1) r ≤ (1 − ηλ ) · ¯Λ ( t ) r , for r ∈ { +1 , − 1 } ;

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ϵ ) , (cid:101)
Revised: ϵ ) , (cid:101)

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: ϵ n u (cid:17)(cid:27) · (Γ ( t )
Revised: ϵ n u (cid:17)(cid:27) · (Γ ( t )

Original: [Equation]
Revised: [Equation]

Original: According to condition 4.1, assume n u = Ω( d 4 ϵ ) and note that ϵ < 1 / 8 , we have
Revised: According to condition 4.1, assume n u = Ω( d 4 ϵ ) and note that ϵ < 1 / 8 , we have

Original: [Equation]
Revised: [Equation]

Original: leading to
Revised: leading to

Original: [Equation]
Revised: [Equation]

Original: By leveraging tensor power method introduced in Lemma C.2, we can prove following lemma about the magnitude of Γ ( t ) :
Revised: By leveraging tensor power method introduced in Lemma C.2, we can prove following lemma about the magnitude of Γ ( t ) :

Original: Lemma A.14.
Revised: Lemma A.14.

Original: Γ ( t ) remains initialization magnitude during [0 , max r ∈{± 1 } { T r } ] .
Revised: Γ ( t ) remains initialization magnitude during [0 , max r ∈{± 1 } { T r } ] .

Original: Proof of lemma A.14.
Revised: Proof of lemma A.14.

Original: Let T ∗ r be the first iteration t in which (cid:98) Λ ( t ) r ≥ A for r ∈ { +1 , − 1 } , let T ∗ be the first iteration t in which Γ ( t ) ≥ A ′ , then according to lemma C.2, we know
Revised: Let T ∗ r be the first iteration t in which (cid:98) Λ ( t ) r ≥ A for r ∈ { +1 , − 1 } , let T ∗ be the first iteration t in which Γ ( t ) ≥ A ′ , then according to lemma C.2, we know

Original: [Equation]
Revised: [Equation]

Original: And it follows that
Revised: And it follows that

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 = (2 p − 1) · (cid:101) Θ( d ) and C ′ 1 , C ′ 

Revised: where C 1 , C 2 = (2 p − 1) · (cid:101) Θ( d ) and C ′ 1 , C ′

Original: = (cid:101)
Revised: = (cid:101)

Original: Θ( d 1 − 2 ϵ ) according to (A.25).
Revised: Θ( d 1 − 2 ϵ ) according to (A.25).

Original: Taking A = Θ(1 /m ) , A ′ =
Revised: Taking A = Θ(1 /m ) , A ′ =

Original: C · Γ ( t ) where C is a large constant and C = Θ(1) , δ = δ ′
Revised: C · Γ ( t ) where C is a large constant and C = Θ(1) , δ = δ ′

Original: = 12 and note that (cid:98) Λ (0)
Revised: = 12 and note that (cid:98) Λ (0)

Original: r = (cid:101) Θ( σ 0 d 12 )
Revised: r = (cid:101) Θ( σ 0 d 12 )

Original: = (cid:101) Θ( d − 14 ) , Γ (0)
Revised: = (cid:101) Θ( d − 14 ) , Γ (0)

Original: = (cid:101) Θ( σ 

Revised: = (cid:101) Θ( σ

Original: σ p d 12 )
Revised: σ p d 12 )

Original: = (cid:101) Θ( d − 14 + ϵ ) , we have η · T ∗ r ≤ (cid:101) Θ( d − 34 ) + η · (cid:101) Θ(1) =
Revised: = (cid:101) Θ( d − 14 + ϵ ) , we have η · T ∗ r ≤ (cid:101) Θ( d − 34 ) + η · (cid:101) Θ(1) =

Original: (cid:101) Θ( d − 34 ) , (A.26) and η · T ∗
Revised: (cid:101) Θ( d − 34 ) , (A.26) and η · T ∗

Original: [Equation]
Revised: [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: 34 + ϵ ) .
Revised: 34 + ϵ ) .

Original: (A.27)
Revised: (A.27)

Original: Therefore, combining (A.26) and (A.27), we have η · T ∗ ≥ (cid:101)
Revised: Therefore, combining (A.26) and (A.27), we have η · T ∗ ≥ (cid:101)

Original: Θ( d − 34 + ϵ ) > (cid:101)
Revised: Θ( d − 34 + ϵ ) > (cid:101)

Original: Θ( d − 34 ) ≥ η · T ∗ r , leading to T ∗ > T ∗ r for both r ∈ {− 1 .
Revised: Θ( d − 34 ) ≥ η · T ∗ r , leading to T ∗ > T ∗ r for both r ∈ {− 1 .

Original: + 1 } .
Revised: + 1 } .

Original: This indicates that when (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 reach Θ(1 /m ) , Γ ( t ) remain the same magnitude as initialization.
Revised: This indicates that when (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t ) − 1 reach Θ(1 /m ) , Γ ( t ) remain the same magnitude as initialization.

Original: By leveraging tensor power method, we can also estimate the length of Stage I, i.e. T 1 , T − 1 , by applying tensor power method.
Revised: By leveraging tensor power method, we can also estimate the length of Stage I, i.e. T 1 , T − 1 , by applying tensor power method.

Original: To use tensor power method, we need to upper-bound the increasing speed of (cid:98) Λ ( t ) r .
Revised: To use tensor power method, we need to upper-bound the increasing speed of (cid:98) Λ ( t ) r .

Original: We have the following lemma:
Revised: We have the following lemma:

Original: Lemma A.15.
Revised: Lemma A.15.

Original: For r ∈ {± 1 } , we have with high probability that
Revised: For r ∈ {± 1 } , we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma A.15.
Revised: Proof of lemma A.15.

Original: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)
Revised: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)

Original: j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ]
Revised: j , v ⟩ and note that u j ∗ = 1 [1 ≤ j ≤ m ]

Original: =[
Revised: =[

Original: m +1 ≤ j ≤ 2 m ]
Revised: m +1 ≤ j ≤ 2 m ]

Original: = 1 , then we have
Revised: = 1 , then we have

Original: [Equation]
Revised: [Equation]

Original: Λ ( t +1)= ⟨ w ( t +1)
Revised: Λ ( t +1)= ⟨ w ( t +1)

Original: j ∗ , v ⟩
Revised: j ∗ , v ⟩

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ♣
Revised: ♣

Original: [Equation]
Revised: [Equation]

Original: (A.28)
Revised: (A.28)

Original: For ♣ , according to lemma A.8, we have
Revised: For ♣ , according to lemma A.8, we have

Original: [Equation]
Revised: [Equation]

Original: = n
Revised: = n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ n
Revised: ≤ n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to (cid:98) Λ ( t )
Revised: where the last equality is due to (cid:98) Λ ( t )

Original: 1 = ω (¯Λ ( t ) − 1 ) .
Revised: 1 = ω (¯Λ ( t ) − 1 ) .

Original: For ⋆ , according to lemma A.7, we have
Revised: For ⋆ , according to lemma A.7, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to (cid:98) Λ ( t )
Revised: where the last equality is due to (cid:98) Λ ( t )

Original: 1 = ω ( ¯Λ ( t ) − 1 )
Revised: 1 = ω ( ¯Λ ( t ) − 1 )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that we have already proved in (A.9)
Revised: Note that we have already proved in (A.9)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Note we have already prove in (A.10)
Revised: Note we have already prove in (A.10)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: In a similar way
Revised: In a similar way

Original: [Equation]
Revised: [Equation]

Original: we can prove that
Revised: we can prove that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of this lemma.
Revised: which completes the proof of this lemma.

Original: By leveraging tensor power method given in lemma C.2,
Revised: By leveraging tensor power method given in lemma C.2,

Original: [Equation]
Revised: [Equation]

Original: we have for r ∈ {± 1 } that
Revised: we have for r ∈ {± 1 } that

Original: [Equation]
Revised: [Equation]

Original: Λ (0) r C(cid:124) (cid:123)(cid:122) (cid:125) ( i ) + η · C 2 C 1 (1 + δ ) q − 1 (cid:18) 1 + log ( A/ (cid:98) Λ (0) r ) log (1 + δ ) (cid:19) (cid:124) (cid:123)(cid:122) (cid:125) ( ii ) , η · T ∗ r = (cid:88) t ≥ 0 , (cid:98) Λ ( t ) r ≤
Revised: Λ (0) r C(cid:124) (cid:123)(cid:122) (cid:125) ( i ) + η · C 2 C 1 (1 + δ ) q − 1 (cid:18) 1 + log ( A/ (cid:98) Λ (0) r ) log (1 + δ ) (cid:19) (cid:124) (cid:123)(cid:122) (cid:125) ( ii ) , η · T ∗ r = (cid:88) t ≥ 0 , (cid:98) Λ ( t ) r ≤

Original: A η ≥ δ (cid:0) 1 − ( x 0 /A )
Revised: A η ≥ δ (cid:0) 1 − ( x 0 /A )

Original: q − 2 (cid:1) (1 + δ ) q − 1 (cid:0) 1 − (1 + δ ) − ( q − 2) (cid:1)(cid:98) Λ (0) r C(cid:124) (cid:123)(cid:122) (cid:125) ( iii ) − η · (1 + δ ) − ( q − 1) (cid:18) 1 + log ( A/ (cid:98) Λ (0) r ) log (1 + δ ) (cid:19) (cid:124) (cid:123)(cid:122) (cid:125) ( iv ) , where C 1 is taken as q (cid:0) p − 12 − o (1) (cid:1) · ∥ v ∥ 22 /m and C 2 is taken as q (cid:0) p − 

Revised: q − 2 (cid:1) (1 + δ ) q − 1 (cid:0) 1 − (1 + δ ) − ( q − 2) (cid:1)(cid:98) Λ (0) r C(cid:124) (cid:123)(cid:122) (cid:125) ( iii ) − η · (1 + δ ) − ( q − 1) (cid:18) 1 + log ( A/ (cid:98) Λ (0) r ) log (1 + δ ) (cid:19) (cid:124) (cid:123)(cid:122) (cid:125) ( iv ) , where C 1 is taken as q (cid:0) p − 12 − o (1) (cid:1) · ∥ v ∥ 22 /m and C 2 is taken as q (cid:0) p −

Original: + o (1) (cid:1) · ∥ v ∥ 22 /m
Revised: + o (1) (cid:1) · ∥ v ∥ 22 /m

Original: according to lemma A.15.
Revised: according to lemma A.15.

Original: Taking δ = 1 k , A = Θ(1 /m ) and note that terms ( ii ) , ( iv ) are respectively dominated by terms ( i ) , ( iii ) when η is sufficiently small and letting k → ∞ , we have
Revised: Taking δ = 1 k , A = Θ(1 /m ) and note that terms ( ii ) , ( iv ) are respectively dominated by terms ( i ) , ( iii ) when η is sufficiently small and letting k → ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: lower order
Revised: lower order

Original: [Equation]
Revised: [Equation]

Original: , for r ∈ { +1 , − 1 } .
Revised: , for r ∈ { +1 , − 1 } .

Original: And it follows that
Revised: And it follows that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.16 (Training time of pretext task) .
Revised: Lemma A.16 (Training time of pretext task) .

Original: For r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) respectively.
Revised: For r ∈ {± 1 } , let T r be the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) respectively.

Original: Then T r = (cid:101) Θ( d − 34 ) /η for all r ∈ {± 1 } .
Revised: Then T r = (cid:101) Θ( d − 34 ) /η for all r ∈ {± 1 } .

Original: The discussion above verifies lemma A.16 and the latter part of lemma 5.3 and provides a clear understanding about how (cid:98) Λ ( t ) r , ¯Λ ( t ) r varies within the iteration range [0 , T r ] for r ∈ { +1 , − 1 } .
Revised: The discussion above verifies lemma A.16 and the latter part of lemma 5.3 and provides a clear understanding about how (cid:98) Λ ( t ) r , ¯Λ ( t ) r varies within the iteration range [0 , T r ] for r ∈ { +1 , − 1 } .

Original: Note that the iteration numbers when (cid:98) Λ ( t ) 1 and (cid:98) Λ ( t ) − 1 reaches Θ(1 /m )
Revised: Note that the iteration numbers when (cid:98) Λ ( t ) 1 and (cid:98) Λ ( t ) − 1 reaches Θ(1 /m )

Original: ( T 1 and T − 1 ) are different.
Revised: ( T 1 and T − 1 ) are different.

Original: However, since T − 1 and T 1 have the same magnitude, it remains clear that although T 1 ̸ = T −(wlog, assume T 1 < T − 1 ), we still have (cid:98) Λ ( t )
Revised: However, since T − 1 and T 1 have the same magnitude, it remains clear that although T 1 ̸ = T −(wlog, assume T 1 < T − 1 ), we still have (cid:98) Λ ( t )

Original: [Equation]
Revised: [Equation]

Original: 1) and ¯Λ ( t )= (cid:101) O
Revised: 1) and ¯Λ ( t )= (cid:101) O

Original: ( d − 14 ) within the iteration range [ T 1 , T − 1 ] , since off-diagonal feature learning also costs time no less than order Θ(1 /ησ 

Revised: ( d − 14 ) within the iteration range [ T 1 , T − 1 ] , since off-diagonal feature learning also costs time no less than order Θ(1 /ησ

Original: ∥ v ∥ 3 √ log m ) , which is higher order than | T 1 − T − 1 | = Θ(1 /ησ 

Revised: ∥ v ∥ 3 √ log m ) , which is higher order than | T 1 − T − 1 | = Θ(1 /ησ

Original: ∥ v ∥ 32 log m ) , according to lemma A.3 and (A.33).
Revised: ∥ v ∥ 32 log m ) , according to lemma A.3 and (A.33).

Original: Therefore, at time T 0 := max { T 1 , T − 1 } , off-diagonal ¯Λ ( t ) 1 , ¯Λ ( t ) −still remain initialization magnitude (cid:101)
Revised: Therefore, at time T 0 := max { T 1 , T − 1 } , off-diagonal ¯Λ ( t ) 1 , ¯Λ ( t ) −still remain initialization magnitude (cid:101)

Original: O
Revised: O

Original: ( d − 14 ) , Γ ( t ) 1 , Γ ( t ) − 1 remain initialization magnitude (cid:101)
Revised: ( d − 14 ) , Γ ( t ) 1 , Γ ( t ) − 1 remain initialization magnitude (cid:101)

Original: O
Revised: O

Original: ( d − 14 + ϵ ) , while on-diagonal (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t )
Revised: ( d − 14 + ϵ ) , while on-diagonal (cid:98) Λ ( t ) 1 , (cid:98) Λ ( t )

Original: − 1 reach and then remain (cid:101) Θ(1) .
Revised: − 1 reach and then remain (cid:101) Θ(1) .

Original: A. P ROOF OF LEMMA 5. 
Revised: A. P ROOF OF LEMMA 5.

Original: If we only use labeled data S ′ for the optimization of CNN, according to lemma B.1, we have w ( t +1)
Revised: If we only use labeled data S ′ for the optimization of CNN, according to lemma B.1, we have w ( t +1)

Original: j
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: i (cid:1) , where u j := 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ 2 m ] , b ( t ) i = − ℓ ′ ( y ′ i · f W ( x ′ i )) =
Revised: i (cid:1) , where u j := 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ 2 m ] , b ( t ) i = − ℓ ′ ( y ′ i · f W ( x ′ i )) =

Original: exp[ − y ′
Revised: exp[ − y ′

Original: i
Revised: i

Original: · f W ( x ′ i )] /
Revised: · f W ( x ′ i )] /

Original: (1+exp[ − y ′
Revised: (1+exp[ − y ′

Original: i · f W ( x ′ i )]) .
Revised: i · f W ( x ′ i )]) .

Original: Notice that v and ξ ′
Revised: Notice that v and ξ ′

Original: i are orthogonal to each other, we have ⟨ w ( t +1)
Revised: i are orthogonal to each other, we have ⟨ w ( t +1)

Original: j ,
Revised: j ,

Original: [Equation]
Revised: [Equation]

Original: Let T ′
Revised: Let T ′

Original: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) , then we have following lemma:
Revised: i be the first iteration that Γ ′ ( t ) i reaches Θ(1 /m ) , then we have following lemma:

Original: Lemma A.17.
Revised: Lemma A.17.

Original: As long as Γ ′ ( t ) i ≤ Θ(1 /m ) , b ( t ) i := − ℓ ′ ( y ′ i · f W ( t ) ( x ′ i )) will remain 1 / 2 ±
Revised: As long as Γ ′ ( t ) i ≤ Θ(1 /m ) , b ( t ) i := − ℓ ′ ( y ′ i · f W ( t ) ( x ′ i )) will remain 1 / 2 ±

Original: o (1) .
Revised: o (1) .

Original: Proof of lemma A.17.
Revised: Proof of lemma A.17.

Original: Note that ℓ ( z ) = log(1+exp ( − z )) and − ℓ ′ ( z ) =
Revised: Note that ℓ ( z ) = log(1+exp ( − z )) and − ℓ ′ ( z ) =

Original: exp ( − z ) / (cid:0) 1+exp ( − z ) (cid:1) , and without loss of generality assuming y ′ i
Revised: exp ( − z ) / (cid:0) 1+exp ( − z ) (cid:1) , and without loss of generality assuming y ′ i

Original: = 1 , we can express b ( t ) i as follow:
Revised: = 1 , we can express b ( t ) i as follow:

Original: [Equation]
Revised: [Equation]

Original: Since σ ( ⟨ w ( t ) j , ξ ⟩ ) will dominate σ ( ⟨ w ( t ) j , v ⟩ ) , which will be proved later by using tensor power method , we have
Revised: Since σ ( ⟨ w ( t ) j , ξ ⟩ ) will dominate σ ( ⟨ w ( t ) j , v ⟩ ) , which will be proved later by using tensor power method , we have

Original: [Equation]
Revised: [Equation]

Original: On the one side,
Revised: On the one side,

Original: [Equation]
Revised: [Equation]

Original: On the other side, according to lemma 5.4, we have ¯Λ ( t ) 

Revised: On the other side, according to lemma 5.4, we have ¯Λ ( t )

Original: = (cid:101) O
Revised: = (cid:101) O

Original: ( d − 14 ) , it follows that
Revised: ( d − 14 ) , it follows that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have b ( t ) i
Revised: we have b ( t ) i

Original: [Equation]
Revised: [Equation]

Original: and the other case of y i = − 1 can be proved in a similar way.
Revised: and the other case of y i = − 1 can be proved in a similar way.

Original: With the help of above lemma, we are now ready to prove lemma 5.2.
Revised: With the help of above lemma, we are now ready to prove lemma 5.2.

Original: Proof of lemma 5.2.
Revised: Proof of lemma 5.2.

Original: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)
Revised: Let j ∗ = arg max 1 ≤ j ≤ m ⟨ w ( t +1)

Original: j , v ⟩ and note that u j = 1 , according to lemma
Revised: j , v ⟩ and note that u j = 1 , according to lemma

Original: A.17, we have
Revised: A.17, we have

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: A.

Original: [Equation]
Revised: [Equation]

Original: For ♣ , we have (cid:88) i ∈
Revised: For ♣ , we have (cid:88) i ∈

Original: S ′(cid:16)2 ± o
Revised: S ′(cid:16)2 ± o

Original: [Equation]
Revised: [Equation]

Original: For ⋆ , we have
Revised: For ⋆ , we have

Original: [Equation]
Revised: [Equation]

Original: By plugging (A.35) and (A.36) in (A.34) [Equation] and according to lemma A.
Revised: By plugging (A.35) and (A.36) in (A.34), and according to lemma A.

Original: [Equation]
Revised: [Equation]

Original: we have with probability at least 1 − 4 δ that
Revised: we have with probability at least 1 − 4 δ that

Original: [Equation]
Revised: [Equation]

Original: And we can prove in the same way that with probability at least 1 − 4 δ we have
Revised: And we can prove in the same way that with probability at least 1 − 4 δ we have

Original: [Equation]
Revised: [Equation]

Original: Let j ⋆ = arg max m +1 ≤ j ≤ 2 m ⟨ w ( t +1)
Revised: Let j ⋆ = arg max m +1 ≤ j ≤ 2 m ⟨ w ( t +1)

Original: j , v ⟩ and note that u j = − 1 , we have ¯Λ ( t +1)
Revised: j , v ⟩ and note that u j = − 1 , we have ¯Λ [Equation] t + [Equation]

Revised: ⋆

Original: [Equation]
Revised: [Equation]

Original: And we can prove in the same way that ¯Λ ( t +1) −
Revised: And we can prove in the same way that ¯Λ ( t +1) −

Original: [Equation]
Revised: [Equation]

Original: Next, we consider the increasing rate of Γ ′ ( t ) l where l ∈ [ n l ] is fixed.
Revised: Next, we consider the increasing rate of Γ ′ ( t ) l where l ∈ [ n l ] is fixed.

Original: If y l = 1 , let j ♮ =
Revised: If y l = 1 , let j ♮ =

Original: argmax 1 ≤ j ≤ m ⟨ w ( t ) j , ξ ′ l ⟩ and note that u j = 1 , we have 
Revised: argmax 1 ≤ j ≤ m ⟨ w ( t ) j , ξ ′ l ⟩ and note that u j = 1 , we have Γ ′ [Equation] t

Original: [Equation]
Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: [Equation]

Original: ± lower order terms }
Revised: ± lower order terms }

Original: [Equation]
Revised: [Equation]

Original: where the third equality holds if we properly choose the order of λ .
Revised: where the third equality holds if we properly choose the order of λ .

Original: If y l = − 1 , let j ♯ = argmax m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , ξ ′ l ⟩ and note that u j = − 1 , we have Γ ′ [Equation] + [Equation]
Revised: If y l = − 1 , let j ♯ = argmax m +1 ≤ j ≤ 2 m ⟨ w ( t ) j , ξ ′ l ⟩ and note that u j = − 1 , we have Γ ′ ( t +1)

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: terms

Original: [Equation]
Revised: [Equation]

Original: where the third equality holds if we properly choose the order of λ .
Revised: where the third equality holds if we properly choose the order of λ .

Original: According to (A.38) and (A.39), we always have
Revised: According to (A.38) and (A.39), we always have

Original: [Equation]
Revised: [Equation]

Original: Θ( d 1+2 ϵ ) · (Γ ′ ( t ) l ) q − 1 .
Revised: Θ( d 1+2 ϵ ) · (Γ ′ ( t ) l ) q − 1 .

Original: A. P ROOF OF LEMMA 5. 
Revised: A. P ROOF OF LEMMA 5.

Original: By applying Lemma C.2 to Γ ( t ) i and taking C 1 = (cid:101) Θ( d 1+2 ϵ ) , δ = 1 / 2 , A = Θ(1 /m ) , we have (cid:88) t ≥ 0 , Γ ( t ) i ≤ A η ≤ Θ(1 /C 1 Γ ( t ) i ) =
Revised: By applying Lemma C.2 to Γ ( t ) i and taking C 1 = (cid:101) Θ( d 1+2 ϵ ) , δ = 1 / 2 , A = Θ(1 /m ) , we have (cid:88) t ≥ 0 , Γ ( t ) i ≤ A η ≤ Θ(1 /C 1 Γ ( t ) i ) =

Original: (cid:101) Θ( d − 34 − 3 ϵ ) .
Revised: (cid:101) Θ( d − 34 − 3 ϵ ) .

Original: And note the definition of T
Revised: And note the definition of T

Original: ′ i , we have η · T ′ i
Revised: ′ i , we have η · T ′ i

Original: =
Revised: =

Original: (cid:101) Θ( d − 34 − 3 ϵ ) .
Revised: (cid:101) Θ( d − 34 − 3 ϵ ) .

Original: (A.40)
Revised: (A.40)

Original: In Lemma 5.2, we have already prove that
Revised: In Lemma 5.2, we have already prove that

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: − ηλ

Original: [Equation]
Revised: [Equation]

Original: · (cid:98) Λ
Revised: Λ

Original: [Equation]
Revised: [Equation]

Original: t ) r +η · Θ( d
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Λ
Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original: t
Revised: ¯Λ

Original: [Equation]
Revised: [Equation]

Original: q
Revised: t

Original: [Equation]
Revised: [Equation]

Original: 1 + (¯Λ ( t ) 
Revised: r ) q

Original: [Equation]
Revised: [Equation]

Original: r ) q − 1 (cid:17) , (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t +1) r , r ∈ {± 1 } .
Revised: 1 (cid:17) , (cid:98) Λ ( t +1) r ≤ (1 − ηλ ) · (cid:98) Λ ( t +1) r , r ∈ {± 1 } .

Original: (A.41)
Revised: (A.41)

Original: Define Λ ( t ) :=
Revised: Define Λ ( t ) :=

Original: max r ∈{± 1 } { (cid:98) Λ ( t ) r , ¯Λ ( t ) r } , according to (A.41), we have Λ ( t +1) ≤ (1 − ηλ ) · Λ ( t ) + η · Θ( d ) · (Λ ( t ) )
Revised: max r ∈{± 1 } { (cid:98) Λ ( t ) r , ¯Λ ( t ) r } , according to (A.41), we have Λ ( t +1) ≤ (1 − ηλ ) · Λ ( t ) + η · Θ( d ) · (Λ ( t ) )

Original: q − 1 .
Revised: q − 1 .

Original: By applying Lemma C.2 to Λ ( t ) , and taking C 1 = Θ( d ) , δ = 1 / 2 , A = C · Λ (0) , where A is a large constant, we have (cid:88)t ≥ 0 , Λ ( t ) ≤ A η [Equation] [Equation]
Revised: By applying Lemma C.2 to Λ ( t ) , and taking C 1 = Θ( d ) , δ = 1 / 2 , A = C · Λ (0) , where A is a large constant, we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:101) Θ( d 14 ) .

Original: Let T ′ be the first iteration that Λ ( t ) reaches C · Λ (0) , then we have η
Revised: Let T ′ be the first iteration that Λ ( t ) reaches C · Λ (0) , then we have η

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: According to (A.40) and (A.42), we have T ′ =
Revised: According to (A.40) and (A.42), we have T ′ =

Original: ω
Revised: ω

Original: ( T ′ i ) , which indicates that when Γ ( t ) i reaches Θ(1 /m ) , Λ ( t ) remains initialization magnitude (cid:101) Θ( d − 1 4 ) .
Revised: ( T ′ i ) , which indicates that when Γ ( t ) i reaches Θ(1 /m ) , Λ ( t ) remains initialization magnitude (cid:101) Θ( d − 1 4 ) .

Original: A. E MPIRICAL , TEST ERROR AND LOSS FOR EARLY STOPPED CLASSIFIER
Revised: A. E MPIRICAL , TEST ERROR AND LOSS FOR EARLY STOPPED CLASSIFIER

Original: Assume the accuracy of pseudolabeler p is larger than 1 / 2 .
Revised: Assume the accuracy of pseudolabeler p is larger than 1 / 2 .

Original: We first estimate the empirical loss for early stopped classifier f W ( T 0) , where T 0 = max r ∈{± 1 } { T r } and T r is defined as the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) .
Revised: We first estimate the empirical loss for early stopped classifier f W ( T 0) , where T 0 = max r ∈{± 1 } { T r } and T r is defined as the first iteration that (cid:98) Λ ( t ) r reaches Θ(1 /m ) .

Original: According to section A.4.3 and lemma A.14, we have (cid:98) Λ ( T 0 ) r
Revised: According to section A.4.3 and lemma A.14, we have (cid:98) Λ ( T 0 ) r

Original: = (cid:101) Θ(1) , ¯Λ ( T 0 )
Revised: = (cid:101) Θ(1) , ¯Λ ( T 0 )

Original: r = (cid:101)
Revised: r = (cid:101)

Original: O
Revised: O

Original: ( d − 14 ) , Γ ( t ) =
Revised: ( d − 14 ) , Γ ( t ) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: ( d − 14 + ϵ ) , for r ∈ { +1 , − 1 } .
Revised: ( d − 14 + ϵ ) , for r ∈ { +1 , − 1 } .

Original: We have the following lemma:
Revised: We have the following lemma:

Original: Lemma A.18.
Revised: Lemma A.18.

Original: Early stopped classifier f W ( T 0) ( x ) possesses following properties:
Revised: Early stopped classifier f W ( T 0) ( x ) possesses following properties:

Original: Training error of early stopped classifier f W ( T 0) ( x ) is asymptotically 1 − p :
Revised: Training error of early stopped classifier f W ( T 0) ( x ) is asymptotically 1 − p :

Original: [Equation]
Revised: [Equation]

Original: Test error is nearly 1 − p , if we use pseudolabel (cid:98) y generated by pseudolabeler as target: P ( x ,y ) ∼D , (cid:98) y ∼ y
Revised: Test error is nearly 1 − p , if we use pseudolabel (cid:98) y generated by pseudolabeler as target: P ( x ,y ) ∼D , (cid:98) y ∼ y

Original: ·B ( p ) [ (cid:98)
Revised: ·B ( p ) [ (cid:98)

Original: y · f W ( T 0) ( x ) ≤ 0] = 1 − p ±
Revised: y · f W ( T 0) ( x ) ≤ 0] = 1 − p ±

Original: o (1) .
Revised: o (1) .

Original: Test error is nearly 0 , if we use true label y as target: P ( x ,y )
Revised: Test error is nearly 0 , if we use true label y as target: P ( x ,y )

Original: ∼D [ y · f W ( T 0) ( x ) ≤ 0]
Revised: ∼D [ y · f W ( T 0) ( x ) ≤ 0]

Original: = o (1) and hence sign f W ( T 0)
Revised: = o (1) and hence sign f W ( T 0)

Original: ( x ) = sign( y ) with high probability,
Revised: ( x ) = sign( y ) with high probability,

Original: where p is the accuracy of the pseudolabeler.
Revised: where p is the accuracy of the pseudolabeler.

Original: We can regard p as the probability that x i is paired with true label y i , 1 − p is the probability that x i is paired with wrong label − y i .
Revised: We can regard p as the probability that x i is paired with true label y i , 1 − p is the probability that x i is paired with wrong label − y i .

Original: Proof of lemma A.18.
Revised: Proof of lemma A.18.

Original: Recall the definition of f W in (3.1) that
Revised: Recall the definition of f W in (3.1) that

Original: [Equation]
Revised: [Equation]

Original: According to section A.4.3 and lemma A.14, we have (cid:98) Λ ( T 0 ) r
Revised: According to section A.4.3 and lemma A.14, we have (cid:98) Λ ( T 0 ) r

Original: = (cid:101) Θ(1) , ¯Λ ( T 0 )
Revised: = (cid:101) Θ(1) , ¯Λ ( T 0 )

Original: r = (cid:101)
Revised: r = (cid:101)

Original: O
Revised: O

Original: ( d − 14 ) , Γ ( t ) = max (cid:8) max i ∈ [ n u ] Γ ( t ) i , max i ∈ [ n l ] Γ ′ ( t ) i (cid:9)
Revised: ( d − 14 ) , Γ ( t ) = max (cid:8) max i ∈ [ n u ] Γ ( t ) i , max i ∈ [ n l ] Γ ′ ( t ) i (cid:9)

Original: =
Revised: =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( d − 14 + ϵ ) , for r ∈ { +1 , − 1 } .
Revised: ( d − 14 + ϵ ) , for r ∈ { +1 , − 1 } .

Original: If y i = 1 , we have following lower bound for f W ( T 0)
Revised: If y i = 1 , we have following lower bound for f W ( T 0)

Original: ( x i ) f W ( T 0) ( x i ) = m
Revised: ( x i ) f W ( T 0) ( x i ) = m

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≥ (cid:0)(cid:98) Λ ( T 0 )(cid:1) q
Revised: ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≥ (cid:0)(cid:98) Λ ( T 0 )(cid:1) q

Original: + (cid:0) Γ ( T 0 ) i (cid:1) q − m (cid:0) ¯Λ ( T 0 )(cid:1) q − m (cid:0) Γ ( T 0 ) i (cid:1) q ≥ (cid:0) (cid:98) Λ ( T 0 )(cid:1) q {− lower order terms } , and following upper bound for f W ( T 0) ( x i ) :
Revised: + (cid:0) Γ ( T 0 ) i (cid:1) q − m (cid:0) ¯Λ ( T 0 )(cid:1) q − m (cid:0) Γ ( T 0 ) i (cid:1) q ≥ (cid:0) (cid:98) Λ ( T 0 )(cid:1) q {− lower order terms } , and following upper bound for f W ( T 0) ( x i ) :

Original: f W ( T 0) ( x i ) = m (cid:88)
Revised: f W ( T 0) ( x i ) = m (cid:88)

Original: j = (cid:104) σ (cid:0) ⟨ w ( T 0 )
Revised: j = (cid:104) σ (cid:0) ⟨ w ( T 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) − 2 m (cid:88)
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) − 2 m (cid:88)

Original: j = m + (cid:104) σ (cid:0) ⟨ w ( T 0 )
Revised: j = m + (cid:104) σ (cid:0) ⟨ w ( T 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≤ m ( (cid:98) Λ ( T 0 ))
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≤ m ( (cid:98) Λ ( T 0 ))

Original: q + m (Γ ( T 0 ) i ) q − (cid:0) ¯Λ ( T 0 )(cid:1) q − (cid:0) Γ ( T 0 )
Revised: q + m (Γ ( T 0 ) i ) q − (cid:0) ¯Λ ( T 0 )(cid:1) q − (cid:0) Γ ( T 0 )

Original: i (cid:1) q ≤ ( (cid:98) Λ ( T 0 ))
Revised: i (cid:1) q ≤ ( (cid:98) Λ ( T 0 ))

Original: q { + lower order terms } .
Revised: q { + lower order terms } .

Original: If y i = − 1 , we have following upper bound for f W ( T 0) ( x i ) :
Revised: If y i = − 1 , we have following upper bound for f W ( T 0) ( x i ) :

Original: f W ( T 0) ( x i ) = m (cid:88)
Revised: f W ( T 0) ( x i ) = m (cid:88)

Original: j = (cid:104) σ (cid:0) − ⟨ w ( T 0 )
Revised: j = (cid:104) σ (cid:0) − ⟨ w ( T 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) − 2 m (cid:88)
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) − 2 m (cid:88)

Original: j = m + (cid:104) σ (cid:0) − ⟨ w ( T 0 )
Revised: j = m + (cid:104) σ (cid:0) − ⟨ w ( T 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≤ m (cid:0) ¯Λ ( T 0 ) − 1 (cid:1) q
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T 0 ) j , ξ i ⟩ (cid:1)(cid:105) ≤ m (cid:0) ¯Λ ( T 0 ) − 1 (cid:1) q

Original: + m (cid:0) Γ ( T 0 ) i (cid:1) q − (cid:0)(cid:98) Λ ( T 0 ) − 1 (cid:1) q − (cid:0) Γ ( T 0 ) i (cid:1)
Revised: + m (cid:0) Γ ( T 0 ) i (cid:1) q − (cid:0)(cid:98) Λ ( T 0 ) − 1 (cid:1) q − (cid:0) Γ ( T 0 ) i (cid:1)

Original: q ≤ − (cid:0)(cid:98) Λ ( T 0 ) − 1 (cid:1) q { + lower order terms } ,
Revised: q ≤ − (cid:0)(cid:98) Λ ( T 0 ) − 1 (cid:1) q { + lower order terms } ,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Therefore, for unlabeled data, we have y i · f W ( T 0) ( x i ) ∈ (cid:2)(cid:0) 1 − o (1) (cid:1) · ( (cid:98) Λ ( T 0 )
Revised: Therefore, for unlabeled data, we have y i · f W ( T 0) ( x i ) ∈ (cid:2)(cid:0) 1 − o (1) (cid:1) · ( (cid:98) Λ ( T 0 )

Original: y i )
Revised: y i )

Original: q , (cid:0) m
Revised: q , (cid:0) m

Original: + o (1) (cid:1)
Revised: + o (1) (cid:1)

Original: · ( (cid:98) Λ ( T 0 )
Revised: · ( (cid:98) Λ ( T 0 )

Original: y i ) q (cid:3) and hence sign (cid:0) f W ( T 0) ( x i ) (cid:1) =
Revised: y i ) q (cid:3) and hence sign (cid:0) f W ( T 0) ( x i ) (cid:1) =

Original: sign( y i ) holds with high probability.
Revised: sign( y i ) holds with high probability.

Original: We can also prove for labeled data ( x ′ i , y ′ i ) that y ′ i · f W ( T 0) ( x ′ i ) ∈ (cid:2)(cid:0) 1 − o (1) (cid:1) · ( (cid:98) Λ ( T 0 )
Revised: We can also prove for labeled data ( x ′ i , y ′ i ) that y ′ i · f W ( T 0) ( x ′ i ) ∈ (cid:2)(cid:0) 1 − o (1) (cid:1) · ( (cid:98) Λ ( T 0 )

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: i ) q , (cid:0) m
Revised: i ) q , (cid:0) m

Original: + o
Revised: + o

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: sign
Revised: sign

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: sign( y ′ i ) in the same way.
Revised: sign( y ′ i ) in the same way.

Original: Note that (cid:98)
Revised: Note that (cid:98)

Original: y i takes y i with probability p , − y i with probability p and n l = o ( n u ) , the first statement in this lemma follows obviously.
Revised: y i takes y i with probability p , − y i with probability p and n l = o ( n u ) , the first statement in this lemma follows obviously.

Original: To prove the other two statement, we need to give an upper bound for the norm of w j .
Revised: To prove the other two statement, we need to give an upper bound for the norm of w j .

Original: According to the update rule of w ( t ) j , we have
Revised: According to the update rule of w ( t ) j , we have

Original: [Equation]
Revised: [Equation]

Original: leading to ∥ w ( t +1)
Revised: leading to ∥ w ( t +1)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is by triangle inequality; the second inequality is due to the definition of (cid:98) Λ ( t ) r , ¯Λ ( t ) r , Γ ( t ) , the last inequality is due to lemma 5.3.
Revised: where the first inequality is by triangle inequality; the second inequality is due to the definition of (cid:98) Λ ( t ) r , ¯Λ ( t ) r , Γ ( t ) , the last inequality is due to lemma 5.3.

Original: According to lemma A.16, we know that T r · η = (cid:101) Θ( d − 34 ) , r ∈ {± 1 } and T 0 · η = max r ∈{± 1 } { T r · η } = (cid:101)
Revised: According to lemma A.16, we know that T r · η = (cid:101) Θ( d − 34 ) , r ∈ {± 1 } and T 0 · η = max r ∈{± 1 } { T r · η } = (cid:101)

Original: Θ( d − 34 ) .
Revised: Θ( d − 34 ) .

Original: Note that w (0) j ∼ N ( 0 , σ 20 I d ) , σ 0 =
Revised: Note that w (0) j ∼ N ( 0 , σ 20 I d ) , σ 0 =

Original: Θ( d − 34 ) and hence ∥ w (0) j ∥ 2 = (cid:101) Θ( d − 14 ) , we know that
Revised: Θ( d − 34 ) and hence ∥ w (0) j ∥ 2 = (cid:101) Θ( d − 14 ) , we know that

Original: [Equation]
Revised: [Equation]

Original: = (cid:101) Θ( d − 14 ) .
Revised: = (cid:101) Θ( d − 14 ) .

Original: Therefore, for any ( x , y ) sampled from distribution D where x = [ y · v ⊤ , ξ ⊤ ] ⊤ and ξ ∼ N (0 , σ 2 p ) , we have
Revised: Therefore, for any ( x , y ) sampled from distribution D where x = [ y · v ⊤ , ξ ⊤ ] ⊤ and ξ ∼ N (0 , σ 2 p ) , we have

Original: [Equation]
Revised: [Equation]

Original: And this indicates that ⟨ w ( T 0 ) j , ξ ⟩ will still be dominated by ⟨ w ( T 0 )
Revised: And this indicates that ⟨ w ( T 0 ) j , ξ ⟩ will still be dominated by ⟨ w ( T 0 )

Original: j , v ⟩ , therefore it holds for newly sampled ( x , y ) that
Revised: j , v ⟩ , therefore it holds for newly sampled ( x , y ) that

Original: [Equation]
Revised: [Equation]

Original: which means that
Revised: which means that

Original: [Equation]
Revised: [Equation]

Original: This verifies the third statement that test error is nearly zero.
Revised: This verifies the third statement that test error is nearly zero.

Original: For the second statement, note that
Revised: For the second statement, note that

Original: [Equation]
Revised: [Equation]

Original: which verifies the second statement.
Revised: which verifies the second statement.

Original: A. D OWNSTREAM TASK
Revised: A. D OWNSTREAM TASK

Original: For downstream tasks, we use early stopped classifiers, which are stopped when on-diagonal feature (cid:98) Λ ( t ) r are learned while off-diagonal feature ¯Λ ( t ) r and noise Γ ( t ) are not memorized.
Revised: For downstream tasks, we use early stopped classifiers, which are stopped when on-diagonal feature (cid:98) Λ ( t ) r are learned while off-diagonal feature ¯Λ ( t ) r and noise Γ ( t ) are not memorized.

Original: Assume we have learned K early stopped classifiers f W
Revised: Assume we have learned K early stopped classifiers f W

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ( x ) by using n u pseudo-labeled data generated by pseudolabeler f w1 , · · · , f w K and n l labeled data.
Revised: ( x ) by using n u pseudo-labeled data generated by pseudolabeler f w1 , · · · , f w K and n l labeled data.

Original: Then, we want to design a classifier on the learned representation f W
Revised: Then, we want to design a classifier on the learned representation f W

Original: [Equation]
Revised: [Equation]

Original: to fit y .
Revised: to fit y .

Original: Here we consider training a downstream linear model
Revised: Here we consider training a downstream linear model

Original: [Equation]
Revised: [Equation]

Original: where a k ∈ R denotes the weight as the k -th pretext model.
Revised: where a k ∈ R denotes the weight as the k -th pretext model.

Original: Given labeled training data S ′
Revised: Given labeled training data S ′

Original: = { ( x ′ i , y ′ i ) } n l i =1 , we want to optimize the empirical loss function
Revised: = { ( x ′ i , y ′ i ) } n l i =1 , we want to optimize the empirical loss function

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( z ) =
Revised: where ℓ ( z ) =

Original: log(1+exp( − z )) denotes the cross entropy loss.
Revised: log(1+exp( − z )) denotes the cross entropy loss.

Original: We initialize a as zero and optimize empirical loss function by gradient descent, i.e.
Revised: We initialize a as zero and optimize empirical loss function by gradient descent, i.e.

Original: [Equation]
Revised: [Equation]

Original: In order to estimate the training error and test error for downstream task, we first introduce following lemma about the increasing rate of (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 .
Revised: In order to estimate the training error and test error for downstream task, we first introduce following lemma about the increasing rate of (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 .

Original: Lemma A.19 (Logarithmic increasing rate) .
Revised: Lemma A.19 (Logarithmic increasing rate) .

Original: For any learning rate η > 0 , a ( t ) k will always increase for any k ∈ [ K ] and hence (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 

Revised: For any learning rate η > 0 , a ( t ) k will always increase for any k ∈ [ K ] and hence (cid:13)(cid:13) a ( t ) (cid:13)(cid:13)

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk
Revised: Kk

Original: =1 a ( t ) k .
Revised: =1 a ( t ) k .

Original: And it holds that (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = Θ(log( t )) .
Revised: And it holds that (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = Θ(log( t )) .

Original: In order to give the increasing rate of (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 , we introduce and prove the following lemma:
Revised: In order to give the increasing rate of (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 , we introduce and prove the following lemma:

Original: Lemma A.20.
Revised: Lemma A.20.

Original: Consider following sequence { x t } ∞ t =1 with
Revised: Consider following sequence { x t } ∞ t =1 with

Original: [Equation]
Revised: [Equation]

Original: where a > 1 and C > 0 are constants, and it follows that
Revised: where a > 1 and C > 0 are constants, and it follows that

Original: [Equation]
Revised: [Equation]

Original: a · C · t + 1 (cid:1) ≤ x t ≤ log a (cid:0) ln a · C · t + 1 (cid:1) + C, and x t +1 − x t ≤
Revised: a · C · t + 1 (cid:1) ≤ x t ≤ log a (cid:0) ln a · C · t + 1 (cid:1) + C, and x t +1 − x t ≤

Original: C
Revised: C

Original: C · ln a · t + 1 .
Revised: C · ln a · t + 1 .

Original: Proof of lemma A.20.
Revised: Proof of lemma A.20.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: by adding up above equation from i = 

Revised: by adding up above equation from i =

Original: to i =
Revised: to i =

Original: t − 1 , we have
Revised: t − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where the first arrow is due to a x is monotone increasing.
Revised: where the first arrow is due to a x is monotone increasing.

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: where the first arrow is due to (A.45) and the last arrow is due to a x is monotone increasing.
Revised: where the first arrow is due to (A.45) and the last arrow is due to a x is monotone increasing.

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove lemma A.19.
Revised: Now we are ready to prove lemma A.19.

Original: Proof of lemma A.19.
Revised: Proof of lemma A.19.

Original: Note that we take downstream task linear model
Revised: Note that we take downstream task linear model

Original: g a ( x ) as
Revised: g a ( x ) as

Original: [Equation]
Revised: [Equation]

Original: Then, we have following update rule for model parameter a :
Revised: Then, we have following update rule for model parameter a :

Original: [Equation]
Revised: [Equation]

Original: where we initialize a (0) k as zero for all k ∈
Revised: where we initialize a (0) k as zero for all k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Next, we prove following statement by using induction method: when t ≥ 1 ,
Revised: Next, we prove following statement by using induction method: when t ≥ 1 ,

Original: • a ( t ) k , ∀ k ∈
Revised: • a ( t ) k , ∀ k ∈

Original: [ K ] is non-negative and increasing.
Revised: [ K ] is non-negative and increasing.

Original: • (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 

Revised: • (cid:13)(cid:13) a ( t ) (cid:13)(cid:13)

Original: = (cid:80)
Revised: = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =1 a ( t ) k .
Revised: =1 a ( t ) k .

Original: • a ( t +1) k
Revised: • a ( t +1) k

Original: [Equation]
Revised: [Equation]

Original: Note that a (0) k
Revised: Note that a (0) k

Original: = 0 for all k ∈ [ d ] and therefore g a
Revised: = 0 for all k ∈ [ d ] and therefore g a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k ∈ [ K ] .
Revised: k ∈ [ K ] .

Original: Note that the accuracy of the k -th pseudolabeler p k > 1 / 2 , accoring to the proof of lemma A.18, we have
Revised: Note that the accuracy of the k -th pseudolabeler p k > 1 / 2 , accoring to the proof of lemma A.18, we have

Original: [Equation]
Revised: [Equation]

Original: for all k ∈ [ K ] .
Revised: for all k ∈ [ K ] .

Original: Therefore
Revised: Therefore

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to y
Revised: where the second equality is due to y

Original: ′ i · g a (1) ( x ′ i ) > 0 , exp ( − y ′
Revised: ′ i · g a (1) ( x ′ i ) > 0 , exp ( − y ′

Original: i · g
Revised: i · g

Original: [Equation]
Revised: [Equation]

Original: 1) ( x ′ i )) < 1 and c ∈ (1 / 2 , 1) ; the last equality is due to (A.46).
Revised: 1) ( x ′ i )) < 1 and c ∈ (1 / 2 , 1) ; the last equality is due to (A.46).

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: where c ∈ (1 / 2 , 1) .
Revised: where c ∈ (1 / 2 , 1) .

Original: By then, we have already proved the induction hypothesis of t = 1 .
Revised: By then, we have already proved the induction hypothesis of t = 1 .

Original: Next, assume the induction hypotheses hold for t .
Revised: Next, assume the induction hypotheses hold for t .

Original: For t + 1 , we have
Revised: For t + 1 , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:125) <
Revised: (cid:125) <

Original: · y ′ i f W ( Tk 0 )
Revised: · y ′ i f W ( Tk 0 )

Original: k
Revised: k

Original: ( x ′ i ) (cid:124) (cid:123)(cid:122) (cid:125) >> a ( t ) k > 0 .
Revised: ( x ′ i ) (cid:124) (cid:123)(cid:122) (cid:125) >> a ( t ) k > 0 .

Original: And it follows that ∥ a ( t +1) ∥ 1 =
Revised: And it follows that ∥ a ( t +1) ∥ 1 =

Original: K (cid:88) i = a ( t +1) k and y ′ i · g a ( t +1) ( x ′ i ) =
Revised: K (cid:88) i = a ( t +1) k and y ′ i · g a ( t +1) ( x ′ i ) =

Original: ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) , (A.47) leading to ℓ ′ (cid:0)
Revised: ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) , (A.47) leading to ℓ ′ (cid:0)

Original: y ′ i · g a ( t +1)
Revised: y ′ i · g a ( t +1)

Original: ( x ′ i ) (cid:1) =
Revised: ( x ′ i ) (cid:1) =

Original: − c · (cid:16) exp (cid:0) − ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) (cid:1)(cid:17) , c ∈ (1 / 2 , 1) , and a ( t +2) k
Revised: − c · (cid:16) exp (cid:0) − ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) (cid:1)(cid:17) , c ∈ (1 / 2 , 1) , and a ( t +2) k

Original: = a ( t +1) k + η · (cid:101) Θ(1) · (cid:16) exp (cid:0) − ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) (cid:1)(cid:17) , ∀ k ∈
Revised: = a ( t +1) k + η · (cid:101) Θ(1) · (cid:16) exp (cid:0) − ∥ a ( t +1) ∥ 1 · (cid:101) Θ(1) (cid:1)(cid:17) , ∀ k ∈

Original: [ K ] .
Revised: [ K ] .

Original: This indicates that if induction hypotheses hold for t , then they holds for t + 1 .
Revised: This indicates that if induction hypotheses hold for t , then they holds for t + 1 .

Original: Adding up k ∈ [ K ] , we can obtain ∥
Revised: Adding up k ∈ [ K ] , we can obtain ∥

Original: [Equation]
Revised: [Equation]

Original: (A.48)
Revised: (A.48)

Original: According to lemma A.20, we know that (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = log t/ (cid:101) Θ(1) {± lower order terms w.r.t.
Revised: According to lemma A.20, we know that (cid:13)(cid:13) a ( t ) (cid:13)(cid:13) 1 = log t/ (cid:101) Θ(1) {± lower order terms w.r.t.

Original: t } .
Revised: t } .

Original: The following lemma gives the convergence guarantee of downstream task:
Revised: The following lemma gives the convergence guarantee of downstream task:

Original: Lemma A.21.
Revised: Lemma A.21.

Original: (Convergence Guarantee)
Revised: (Convergence Guarantee)

Original: For any learning rate η > 0 ,
Revised: For any learning rate η > 0 ,

Original: [Equation]
Revised: [Equation]

Original: which means within polynomial steps, gradient descent is guaranteed to find a point with small gradient.
Revised: which means within polynomial steps, gradient descent is guaranteed to find a point with small gradient.

Original: Proof of lemma A.21.
Revised: Proof of lemma A.21.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: then according to lemma A.20 and (A.48)
Revised: then according to lemma A.20 and (A.48)

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: And it follows that
Revised: And it follows that

Original: [Equation]
Revised: [Equation]

Original: which shows that within polynomial steps, gradient descent is guaranteed to find a point with small gradient.
Revised: which shows that within polynomial steps, gradient descent is guaranteed to find a point with small gradient.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: for all k
Revised: for all k

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [ K
Revised: [ K

Original: [Equation]
Revised: [Equation]

Original: Denote (cid:104)
Revised: Denote (cid:104)

Original: [Equation]
Revised: [Equation]

Original: f W ∗ ( x ′ i ) , then
Revised: f W ∗ ( x ′ i ) , then

Original: [Equation]
Revised: [Equation]

Original: Note that f W ∗ ( x ′ i ) · f W ∗ ( x ′ i )
Revised: Note that f W ∗ ( x ′ i ) · f W ∗ ( x ′ i )

Original: ⊤ is a non-negative definite matrix, ℓ ′′ ( z ) =
Revised: ⊤ is a non-negative definite matrix, ℓ ′′ ( z ) =

Original: exp ( − z ) / (cid:0) 1 + exp ( − z ) (cid:1) 2 > 0 and the fact that sum of non-negative definite matrices is still a non-negative definite matrix, it follows that ∇ 2 a L S ( a ) ⪰ 0 .
Revised: exp ( − z ) / (cid:0) 1 + exp ( − z ) (cid:1) 2 > 0 and the fact that sum of non-negative definite matrices is still a non-negative definite matrix, it follows that ∇ 2 a L S ( a ) ⪰ 0 .

Original: Theorem A.22 (Restatement of theorem 4.3) .
Revised: Theorem A.22 (Restatement of theorem 4.3) .

Original: Under self-supervised learning setting, for downstream task, suppose K early stopped classifiers { f W ∗ k } Kk =1 are obtained after the training of K pretext tasks finished, and after T dt = (cid:101) Θ(1) iterations with learning rate η > 0 , then we can find a linear model a ( T dt ) , which satisfies: Both test error and loss are nearly 0, i.e. P ( x ,y )
Revised: Under self-supervised learning setting, for downstream task, suppose K early stopped classifiers { f W ∗ k } Kk =1 are obtained after the training of K pretext tasks finished, and after T dt = (cid:101) Θ(1) iterations with learning rate η > 0 , then we can find a linear model a ( T dt ) , which satisfies: Both test error and loss are nearly 0, i.e. P ( x ,y )

Original: ∼D [ y · g a ( T dt) ( x ) ≤ 0]
Revised: ∼D [ y · g a ( T dt) ( x ) ≤ 0]

Original: = o (1) , L D ( ℓ ( y · g a ( T dt) ( x )))
Revised: = o (1) , L D ( ℓ ( y · g a ( T dt) ( x )))

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: Proof of theorem A.22.
Revised: Proof of theorem A.22.

Original: For test error, we have
Revised: For test error, we have

Original: [Equation]
Revised: [Equation]

Original: = o (1) where the last equality is due to a ( T dt ) k > 0 according to lemma A.19.
Revised: = o (1) where the last equality is due to a ( T dt ) k > 0 according to lemma A.19.

Original: For test loss, we have
Revised: For test loss, we have

Original: [Equation]
Revised: [Equation]

Original: i.e., we estimate for newly generated data ( x , y ) the magnitude of ℓ ( y · g a ( t ) ( x )) .
Revised: i.e., we estimate for newly generated data ( x , y ) the magnitude of ℓ ( y · g a ( t ) ( x )) .

Original: In order to do so, we will first estimate ℓ ( y ′ i · g a ( t ) ( x i )) .
Revised: In order to do so, we will first estimate ℓ ( y ′ i · g a ( t ) ( x i )) .

Original: Then, we will show that ℓ ( y · g a ( t ) ( x )) and ℓ ( y ′ i · g a ( t ) ( x i )) nearly equal to each other.
Revised: Then, we will show that ℓ ( y · g a ( t ) ( x )) and ℓ ( y ′ i · g a ( t ) ( x i )) nearly equal to each other.

Original: According to the update rule of a ( t ) k , we have
Revised: According to the update rule of a ( t ) k , we have

Original: [Equation]
Revised: [Equation]

Original: Adding up the above equation for k ∈ [ K ] , we obtain
Revised: Adding up the above equation for k ∈ [ K ] , we obtain

Original: [Equation]
Revised: [Equation]

Original: And according to A.49, we have
Revised: And according to A.49, we have

Original: [Equation]
Revised: [Equation]

Original: therefore it follows that
Revised: therefore it follows that

Original: [Equation]
Revised: [Equation]

Original: Note that K = Θ(1) and for all k ∈
Revised: Note that K = Θ(1) and for all k ∈

Original: [ K ] we have y ′
Revised: [ K ] we have y ′

Original: i · f W ( Tk 0 ) k
Revised: i · f W ( Tk 0 ) k

Original: ( x ′ i
Revised: ( x ′ i

Original: [Equation]
Revised: [Equation]

Original: = (cid:101) Θ(1) , it follows that
Revised: = (cid:101) Θ(1) , it follows that

Original: [Equation]
Revised: [Equation]

Original: Note that n l = (cid:101) Θ(1) and according to lemma A.10, there exists a positive sample ( x i 1 , y i 1 ) and a negative sample ( x i 2 , y i 2 ) with the property that
Revised: Note that n l = (cid:101) Θ(1) and according to lemma A.10, there exists a positive sample ( x i 1 , y i 1 ) and a negative sample ( x i 2 , y i 2 ) with the property that

Original: [Equation]
Revised: [Equation]

Original: Note that ℓ ( z )
Revised: Note that ℓ ( z )

Original: =
Revised: =

Original: log(1 + exp( − z )) and ℓ ′ ( z )
Revised: log(1 + exp( − z )) and ℓ ′ ( z )

Original: = − exp( − z ) / (cid:0) 1 + exp( − z ) (cid:1) , we know that for z > 0 ,
Revised: = − exp( − z ) / (cid:0) 1 + exp( − z ) (cid:1) , we know that for z > 0 ,

Original: − ℓ ′ ( z ) =
Revised: − ℓ ′ ( z ) =

Original: c · exp( − z ) , ℓ ( z ) < exp( − z ) = − ℓ ′ ( z ) /c, c ∈ (1 / 2 , 1) .
Revised: c · exp( − z ) , ℓ ( z ) < exp( − z ) = − ℓ ′ ( z ) /c, c ∈ (1 / 2 , 1) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Note that ℓ ( z ) is 1-Lipschitz, we have
Revised: Note that ℓ ( z ) is 1-Lipschitz, we have

Original: [Equation]
Revised: [Equation]

Original: If y = 1 , we have
Revised: If y = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: and f W
Revised: and f W

Original: [Equation]
Revised: [Equation]

Original: Tk 0 )
Revised: Tk 0 )

Original: k
Revised: k

Original: ( x )
Revised: ( x )

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: Tk 0 )
Revised: Tk 0 )

Original: k
Revised: k

Original: ( x ′ i 1 )
Revised: ( x ′ i 1 )

Original: = m (cid:88)
Revised: = m (cid:88)

Original: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )
Revised: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 )
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 )

Original: j , ξ ⟩ (cid:1)(cid:105) − 2 m (cid:88)
Revised: j , ξ ⟩ (cid:1)(cid:105) − 2 m (cid:88)

Original: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 )
Revised: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 )
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 )

Original: j , ξ ⟩ (cid:1)(cid:105) − m (cid:88)
Revised: j , ξ ⟩ (cid:1)(cid:105) − m (cid:88)

Original: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )
Revised: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′

Original: i 1 ⟩ (cid:1)(cid:105) + 2 m (cid:88)
Revised: i 1 ⟩ (cid:1)(cid:105) + 2 m (cid:88)

Original: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 )
Revised: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 )

Original: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′
Revised: j , v ⟩ (cid:1) + σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′

Original: i 1 ⟩ (cid:1)(cid:105) = m (cid:88)
Revised: i 1 ⟩ (cid:1)(cid:105) = m (cid:88)

Original: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )
Revised: j = (cid:104) σ (cid:0) ⟨ w ( T k 0 )

Original: j , ξ ⟩ (cid:1) − σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′ i 1 ⟩ (cid:1)(cid:105) + 2 m (cid:88)
Revised: j , ξ ⟩ (cid:1) − σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′ i 1 ⟩ (cid:1)(cid:105) + 2 m (cid:88)

Original: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′
Revised: j = m + (cid:104) σ (cid:0) ⟨ w ( T k 0 ) j , ξ ′

Original: i 1 ⟩ (cid:1) − σ (cid:0) ⟨ w ( T k 0 )
Revised: i 1 ⟩ (cid:1) − σ (cid:0) ⟨ w ( T k 0 )

Original: j , ξ ⟩ (cid:1)(cid:105) =
Revised: j , ξ ⟩ (cid:1)(cid:105) =

Original: (cid:101)
Revised: (cid:101)

Original: O
Revised: O

Original: ( d − 14 + ϵ ) , (A.52) where the last equality is due to (A.44) and lemma 5.3.
Revised: ( d − 14 + ϵ ) , (A.52) where the last equality is due to (A.44) and lemma 5.3.

Original: Plugging (A.52) into (A.51), we have
Revised: Plugging (A.52) into (A.51), we have

Original: [Equation]
Revised: [Equation]

Original: If y = − 1 , we can prove in a similar way that
Revised: If y = − 1 , we can prove in a similar way that

Original: [Equation]
Revised: [Equation]

Original: Plugging (A.53) and (A.54) into (A.50), we have
Revised: Plugging (A.53) and (A.54) into (A.50), we have

Original: [Equation]
Revised: [Equation]

Original: According to lemma A.20 and (A.48), we have ∥ a ( t ) ∥ 1 = log
Revised: According to lemma A.20 and (A.48), we have ∥ a ( t ) ∥ 1 = log

Original: t/ (cid:101) Θ(1) {± lower order terms w.r.t.
Revised: t/ (cid:101) Θ(1) {± lower order terms w.r.t.

Original: t } , therefore
Revised: t } , therefore

Original: [Equation]
Revised: [Equation]

Original: order
Revised: order

Original: [Equation]
Revised: [Equation]

Original: Taking η = Θ(1) and T dt
Revised: Taking η = Θ(1) and T dt

Original: = Θ( d α /η ) where α > 0 is a sufficiently small constant
Revised: = Θ( d α /η ) where α > 0 is a sufficiently small constant

Original: , we know that
Revised: , we know that

Original: [Equation]
Revised: [Equation]

Original: lower order
Revised: lower order

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: B P ROOF OF SUPERVISED LEARNING SETTING
Revised: B P ROOF OF SUPERVISED LEARNING SETTING

Original: Here we prove theorem 4.4.
Revised: Here we prove theorem 4.4.

Original: First, we give following lemma to facilitate the proof.
Revised: First, we give following lemma to facilitate the proof.

Original: Lemma B.1 (Gradient Calculation) .
Revised: Lemma B.1 (Gradient Calculation) .

Original: The gradient of loss function L S ( W ) with respect to weight parameter w j is
Revised: The gradient of loss function L S ( W ) with respect to weight parameter w j is

Original: [Equation]
Revised: [Equation]

Original: where u j := (cid:0) 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] (cid:1) and − ℓ ′ (cid:0)
Revised: where u j := (cid:0) 1 [1 ≤ j ≤ m ] − 1 [ m +1 ≤ j ≤ 2 m ] (cid:1) and − ℓ ′ (cid:0)

Original: y ′ i · f W ( x ′ i ) (cid:1) =
Revised: y ′ i · f W ( x ′ i ) (cid:1) =

Original: exp [ − y ′
Revised: exp [ − y ′

Original: i
Revised: i

Original: · f W ( x ′ i )] / (1 + exp
Revised: · f W ( x ′ i )] / (1 + exp

Original: [ − y ′ i · f W ( x ′ i )]) is denoted as b i .
Revised: [ − y ′ i · f W ( x ′ i )]) is denoted as b i .

Original: Proof of Lemma B.1.
Revised: Proof of Lemma B.1.

Original: When 1 ≤ j ≤ m ,
Revised: When 1 ≤ j ≤ m ,

Original: [Equation]
Revised: [Equation]

Original: and when m + 1 ≤ j ≤ 2 m
Revised: and when m + 1 ≤ j ≤ 2 m

Original: [Equation]
Revised: [Equation]

Original: Combining above two cases, we have
Revised: Combining above two cases, we have

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: Proof of theorem 4.4.
Revised: Proof of theorem 4.4.

Original: Recall the definition of f W in (3.1) that
Revised: Recall the definition of f W in (3.1) that

Original: [Equation]
Revised: [Equation]

Original: Define (cid:101) w j := m 1 / 3 · w j , we have 
Revised: Define (cid:101) w j := m 1 /q · w j , we have f

Original: [Equation]
Revised: [Equation]

Original:  Since the standard deviation of Gaussian initialization of w j is σ 0 and note that (cid:101) w j := m 1 / 3 · w j , the standard deviation of Gaussian initialization of (cid:101) w j is m 1 / 3 σ 0 :
Revised: j [Equation] Since the standard deviation of Gaussian initialization of w j is σ 0 and note that (cid:101) w j := m 1 /q · w j , the standard deviation of Gaussian initialization of (cid:101) w j is m 1 /q σ 0 :

Original: = (cid:101) σ 0 .
Revised: = (cid:101) σ 0 .

Original: On the other hand, note that the update rule of w ( t ) j is w ( t +1)
Revised: On the other hand, note that the update rule of w ( t ) j is w ( t +1)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: t ) j − η · ∇ w j L S ′
Revised: t ) j − η · ∇ w j L S ′

Original: ( W ( t ) ) , and in lemma B.1, we have
Revised: ( W ( t ) ) , and in lemma B.1, we have

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: By plugging w j = m − 1 / 3 · (cid:101) w j into B.1, we have
Revised: By plugging w j = m − 1 /q · (cid:101) w j into B.1, we have

Original: [Equation]
Revised: [Equation]

Original: Assume (cid:101) η
Revised: Assume (cid:101) η

Original: = m − 13 η , we have (cid:101) w ( t +1)
Revised: = m − 1 q η , we have (cid:101) w ( t +1)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: t ) ) .
Revised: t ) ) .

Original: Therefore, our data model and training algorithm is equivalent to the model and algorithm below:
Revised: Therefore, our data model and training algorithm is equivalent to the model and algorithm below:

Original: [Equation]
Revised: [Equation]

Original: and we use gradient decent with learning rate (cid:101) η and cross-entropy loss to optimize such a data model, i.e.[-[Equation] t ) 
 ∼ N-]
Revised: and we use gradient decent with learning rate (cid:101) η and cross-entropy loss to optimize such a data model, i.e.[-[Equation] t ) 
 ∼ N-]

Original: [Equation]

Original: [Equation]

Original: 20 I d ) ,

Original: [Equation]
Revised: [Equation]

Original: w

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( z ) =
Revised: where ℓ ( z ) =

Original: log(1 + exp( − z )) , (cid:101) σ 0 = m 1 / 3 σ 0 .
Revised: log(1+exp( − z )) , (cid:101) σ 0 = m 1 /q σ 0 .

Original: Note that the new model meets the one used in Cao et al. 
Revised: Note that the new model meets the one used in Cao et al. (2022).

Original: To leverage their result, we introduce condition 4.3 from Cao et al.
Revised: To leverage their result, we introduce condition 4.3 from Cao et al.

Original: and verify that the new model meets the new condition.
Revised: and verify that the new model meets the new condition.

Original: Notice that our model has activation function σ ( z ) =

Original: [ z ] 3+ which is a special case of σ ( z ) =

Original: [ z ] q + considered in Cao et al.

Original: for q = 3 .

Original: We have following simplified condition and Theorems,

Original: Condition B.2 (Condition 4.2 in Cao et al.
Revised: Condition B.2 (Condition 4.2 in Cao et al.

Original: Dimension d  = (cid:101) Ω( m 4 n 4  ) .
Revised: Dimension d is sufficiently large that d = (cid:101) Ω( m 2 ∨ [4 / ( q − 2)] n 4 ∨ [(2 q − 2) / ( q − 2)] ) .

Original: Training sample size n and neural network width m satisfy n, m = Ω(polylog
Revised: Training sample size n and neural network width m satisfy n, m = Ω(polylog

Original: ( d )) .
Revised: ( d )) .

Original: Learning rate η satisfies η ≤ (cid:101) O
Revised: Learning rate η satisfies η ≤ (cid:101) O

Original: (min {∥ v ∥ − 2 2 , σ − 2 p
Revised: (min {∥ v ∥ − 2 2 , σ − 2 p

Original: d
Revised: d

Original: − 1 } ) .
Revised: − 1 } ) .

Original: The standard deviation of Gaussian initialization σ 0 is approximately chosen such that (cid:101) O ( nd − 12 ) · min { ( σ p √ d ) − 1 , ∥ v ∥ − 1 2 } ≤ σ 0 ≤ (cid:101) O
Revised: The standard deviation of Gaussian initialization σ 0 is approximately chosen such that (cid:101) O ( nd − 12 ) · min { ( σ p √ d ) − 1 , ∥ v ∥ − 1 2 } ≤ σ 0 ≤ (cid:101) O

Original: ( m − 2  n −  1 ) · min { ( σ p √ d ) − 1 , ∥ v ∥ − 1 2 } .
Revised: ( m − 2 / ( q − 2) n − [1 / ( q − 2)] ∨ 1 ) · min { ( σ p √ d ) − 1 , ∥ v ∥ − 1 2 } .

Original: Theorem B.3 (Theorem 4.4 in Cao et al. (2022)) .
Revised: Theorem B.3 (Theorem 4.4 in Cao et al. (2022)) .

Original: For any ϵ > 0 , let T =
Revised: For any ϵ > 0 , let T =

Original: (cid:101) Θ( η − 1 m · n ( σ p √ d ) − q · σ − 1 0 + η − 1 ϵ − 1 nm q d − 1 σ − 2 p ) .
Revised: (cid:101) Θ( η − 1 m · n ( σ p √ d ) − q · σ − ( q − 2)+ η − 1 ϵ − 1 nm 3 d − 1 σ − 2 p ) .

Original: Under condition B.2, if n − 1 · SNR − q = (cid:101) Ω(1) , SNR = ∥ v ∥ 2 /σ p √ d , then with probability at least 1 − d − 1 , there exists 0 ≤ t ≤ T such that:
Revised: Under condition B.2, if n − 1 · SNR − q = (cid:101) Ω(1) , SNR = ∥ v ∥ 2 /σ p √ d , then with probability at least 1 − d − 1 , there exists 0 ≤ t ≤ T such that:

Original: The training loss converges to δ ,
Revised: The training loss converges to δ ,

Original: i
Revised: i

Original: e., L S ( W ( t ) ) ≤ δ . 2.
Revised: e., L S ( W ( t ) ) ≤ δ . 2.

Original: The trained CNN has a constant order test loss: L D ( W ( t ) )
Revised: The trained CNN has a constant order test loss: L D ( W ( t ) )

Original: =
Revised: =

Original: Θ(1) .
Revised: Θ(1) .

Original: Note that in our setting, m = Θ(polylog( d )) , n l = (cid:101) Θ(1) , ∥ v ∥ 2 = Θ( d 12 ) , (cid:101) σ 0 = m 1 / 3 σ 0 , σ 0 =
Revised: Note that in our setting, m = Θ(polylog( d )) , n l = (cid:101) Θ(1) , ∥ v ∥ 2 = Θ( d 1 2 ) , (cid:101) σ 0 = m 1 /q σ 0 , σ 0 =

Original: Θ( d − 34 ) σ p =
Revised: Θ( d − 34 ) σ p =

Original: Θ( d 0 . 01 ) , (cid:101) η
Revised: Θ( d 0 . 01 ) , (cid:101) η

Original: = m − 13 η and η = O ( d − 1 − 2 ϵ ) , it’s not difficult to verify that condition
Revised: = m − 1 q η and η = O ( d − 1 − 2 ϵ ) , it’s not difficult to verify that condition

Original: B.2 holds.
Revised: B.2 holds.

Original: Besides, SNR = d − 0 .
Revised: Besides, SNR = d − 0 .

Original: 01 , n − 1 · SNR − q
Revised: 01 , n − 1 · SNR − q

Original: =
Revised: =

Original: (cid:101) Θ
Revised: (cid:101) Θ

Original: ( d qϵ ) =
Revised: ( d qϵ ) =

Original: (cid:101) Ω(1) .
Revised: (cid:101) Ω(1) .

Original: Therefore, the conclusion of
Revised: Therefore, the conclusion of

Original: Theorem B.3 holds for
Revised: Theorem B.3 holds for

Original: [Equation]
Revised: [Equation]

Original: C A UXILIARY L EMMAS
Revised: L EMMAS

Original: For the estimation of ∥ ξ i ∥ 22 and ⟨ ξ i , ξ l ⟩ , we introduce following lemma.
Revised: For the estimation of ∥ ξ i ∥ 2 2 and ⟨ ξ i , ξ l ⟩ , we introduce following lemma.

Original: Lemma C.1 (Lemma B.2 in Cao et al.
Revised: Lemma C.1 (Lemma B.2 in Cao et al.

Original: Suppose that δ > 0 and d = Ω(log(4 n/δ )) .
Revised: Suppose that δ > 0 and d = Ω(log(4 n/δ )) .

Original: Then with probability at least 1 − δ ,
Revised: Then with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: for all i, l ∈ [ n ] , i ̸ = l .
Revised: for all i, l ∈ [ n ] , i ̸ = l .

Original: Besides, we introduce following lemma about tensor power method.
Revised: Besides, we introduce following lemma about tensor power method.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Consider an increasing sequence x t ≥ 0 defined as x t +1 = x t + η · C t x q − 1 t , and 
Revised: Consider an increasing sequence x t ≥ 0 defined as x t +1 = x t + η · C t x q − 1 t , and [Equation]

Original: C 1 ≤ C t ≤ C 2 for all t > 0 , then we have for A > x 0 , every δ > 0 , and every η > 0 :
Revised: C [Equation] ≤ C 2 for all t > 0 , then we have for A > x 0 , every δ > 0 , and every η > 0 :

Original: [Equation]
Revised: [Equation]

Original: Proof of lemma C.2.
Revised: Proof of lemma C.2.

Original: For every g = 0 , 1 , 2 , · · · , let τ g be the first iteration such that x t ≥ (1+ δ ) g x 0 .
Revised: For every g = 0 , 1 , 2 , · · · , let τ g be the first iteration such that x t ≥ (1+ δ ) g x 0 .

Original: Let b be the smallest integer such that (1 + δ ) b x 0 ≥ A .
Revised: Let b be the smallest integer such that (1 + δ ) b x 0 ≥ A .

Original: By the definition of τ g , we have x t ∈
Revised: By the definition of τ g , we have x t ∈

Original: [(1 + δ ) g x 0 , (1 + δ )
Revised: [(1 + δ ) g x 0 , [Equation]

Original:  g +1 x 0 ) for all t ∈ [ τ g , τ g +1 ) and x τ g +1 ≥ (1 + δ )
Revised: 1 + δ ) g +1 x 0 ) for all t ∈ [ τ g , τ g +1 ) and x τ g +1 ≥ [Equation] [Equation]

Original: g +1 x 0 , x τ g − 1 < (1 + δ ) g x 0 , leading to (cid:88)
Revised: δ [Equation] [Equation] , leading to (cid:88)

Original: [Equation]
Revised: [Equation]

Original: following lower bound for x τ g +1 − x τ g :
Revised: following lower bound for x τ g +1 − x τ g :

Original: [Equation]
Revised: [Equation]

Original: and following upper bound for x τ g +1 − x τ g :
Revised: and following upper bound for x τ g +1 − x τ g :

Original: [Equation]
Revised: [Equation]

Original: ( q − 1)( g +1)

Original: x q

Original: [Equation] Therefore,(cid:88) [Equation] t
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: These imply that(cid:88) [Equation] (cid:88) t [Equation]
Revised: These imply that

Original: [Equation]

Original: δ

Original: [Equation]
Revised: [Equation]

Original: Recall b is the smallest integer such that (1 + δ ) b x 0 ≥ A , so we can calculate that(cid:88)
Revised: Recall b is the smallest integer such that (1 + δ ) b x 0 ≥ A , so we can calculate that

Original: [Equation]
Revised: [Equation]

Original: and (cid:88)
Revised: and (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to (1 + δ ) b x 0 ≥ A .
Revised: where the last inequality is due to (1 + δ ) b x 0 ≥ A .

Original: Note that (1 + δ ) b − 1 x 0 < A , i.e. b ≤ 1 + log
Revised: Note that (1 + δ ) b − 1 x 0 < A , i.e. b ≤ 1 + log

Original: ( A/x 0 ) log(1+ δ ) , therefore 
Revised: ( A/x 0 ) log [Equation] 1+ δ ) , therefore (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Note that C 1 ≤ C t ≤ C 2 , we have 
Revised: Note that C 1 ≤ C t ≤ C 2 , we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .