Original: D EMYSTIFYING H OW S ELF -S UPERVISED F EATURES I MPROVE T RAINING FROM N OISY L ABELS
Revised: D EMYSTIFYING H OW S ELF -S UPERVISED F EATURES I MPROVE T RAINING FROM N OISY L ABELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The advancement of self
Revised: The advancement of self

Original: -supervised learning
Revised: -supervised learning

Original: SSL) motivates researchers to apply
Revised: SSL) motivates researchers to apply

Original: SSL on other tasks such as learning with noisy labels.
Revised: SSL on other tasks such as learning with noisy labels.

Original: Recent literature indicates that methods built on SSL features can substantially improve the performance of learning with noisy labels.
Revised: Recent literature indicates that methods built on SSL features can substantially improve the performance of learning with noisy labels.

Original: Nonetheless, the deeper reasons why (and how) SSL features beneﬁt the training from noisy labels are less understood.
Revised: Nonetheless, the deeper reasons why (and how) SSL features beneﬁt the training from noisy labels are less understood.

Original: In this paper, we study why and how self-supervised features help networks resist label noise using both theoretical analyses and numerical experiments.
Revised: In this paper, we study why and how self-supervised features help networks resist label noise using both theoretical analyses and numerical experiments.

Original: Our result shows that, given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise.
Revised: Our result shows that, given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise.

Original: Further, we provide insights for how knowledge distilled from SSL features can alleviate the over-ﬁtting problem.
Revised: Further, we provide insights for how knowledge distilled from SSL features can alleviate the over-ﬁtting problem.

Original: We hope our work provides a better understanding for learning with noisy labels from the perspective of self-supervised learning and can potentially serve as a guideline for further research
Revised: We hope our work provides a better understanding for learning with noisy labels from the perspective of self-supervised learning and can potentially serve as a guideline for further research

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks (DNNs) have achieved remarkable performance in many areas including speech recognition (Graves et al., 2013), computer vision (Krizhevsky et al., 2012; Lotter et al., 2016), natural language processing (Zhang & LeCun, 2015) etc .
Revised: Deep Neural Networks (DNNs) have achieved remarkable performance in many areas including speech recognition (Graves et al., 2013), computer vision (Krizhevsky et al., 2012; Lotter et al., 2016), natural language processing (Zhang & LeCun, 2015) etc .

Original: The high-achieving performance often builds on the availability of quality-annotated datasets.
Revised: The high-achieving performance often builds on the availability of quality-annotated datasets.

Original: In real world scenario, data annotation inevitably brings in label noise which degrades the performance of the network, primarily due to
Revised: In real world scenario, data annotation inevitably brings in label noise which degrades the performance of the network, primarily due to

Original: DNNs’ capability in “memorizing” noisy labels (Zhang et al., 2016).
Revised: DNNs’ capability in “memorizing” noisy labels (Zhang et al., 2016).

Original: In the past few years, a number of methods have been proposed to tackle the problem of learning with label noise including robust loss design (Ghosh et al., 2017; Zhang & Sabuncu, 2018; Liu &
Revised: In the past few years, a number of methods have been proposed to tackle the problem of learning with label noise including robust loss design (Ghosh et al., 2017; Zhang & Sabuncu, 2018; Liu &

Original: Guo, 2020), sample selection (Han et al., 2018; Yu et al., 2019; Cheng et al., 2021) and noise transition matrix estimation (Patrini et al., 2017; Zhu et al., 2021b).
Revised: Guo, 2020), sample selection (Han et al., 2018; Yu et al., 2019; Cheng et al., 2021) and noise transition matrix estimation (Patrini et al., 2017; Zhu et al., 2021b).

Original: Among all these methods, arguably the most efﬁcient treatment is to adopt robust losses, since sample selection and noise transition matrix estimation always involve training multiple networks or need multi-stage training.
Revised: Among all these methods, arguably the most efﬁcient treatment is to adopt robust losses, since sample selection and noise transition matrix estimation always involve training multiple networks or need multi-stage training.

Original: Nonetheless, though the designed losses are theoretically proven robust, they often suffer from signiﬁcant performance drop when noise rate is high (Wang et al., 2019; Ma et al., 2020; Cheng et al., 2021;
Revised: Nonetheless, though the designed losses are theoretically proven robust, they often suffer from signiﬁcant performance drop when noise rate is high (Wang et al., 2019; Ma et al., 2020; Cheng et al., 2021;

Original: Zhu et al., 2021a), indicating that the ability of converging to the optimal classiﬁer is also important.
Revised: Zhu et al., 2021a), hinting that the ability of converging to the optimal classiﬁer is also important.

Original: Very recent works (Zheltonozhskii et al., 2021; Nodet et al., 2021; Ghosh & Lan, 2021; Yao et al., 2021; Tan et al., 2021) started applying self-supervised learning to solving the problem of learning from noisy labels.
Revised: Very recent works (Zheltonozhskii et al., 2021; Nodet et al., 2021; Ghosh & Lan, 2021; Yao et al., 2021; Tan et al., 2021) started applying self-supervised learning to solving the problem of learning from noisy labels.

Original: The experiments show that methods built on the self-supervised features can achieve exceptional performance even when the noise rate is high and largely outperform previously reported SOTA approaches.
Revised: The experiments show that methods built on the self-supervised features can achieve exceptional performance even when the noise rate is high and largely outperform previously reported SOTA approaches.

Original: Despite the empirical observations, the reasons why self-supervised features lead to signiﬁcant performance improvement are not well understood.
Revised: Despite the empirical observations, the reasons why self-supervised features lead to signiﬁcant performance improvement are not well understood.

Original: In this paper, we provide theoretical insights to understand how self-supervised features improve classiﬁcation with label noise and perform extensive experiments to support our theory.
Revised: In this paper, we provide theoretical insights to understand how self-supervised features improve classiﬁcation with label noise and perform extensive experiments to support our theory.

Original: Our analysis provides a new understanding on learning with noisy labels from the perspective of self-supervised learning.
Revised: Our analysis provides a new understanding on learning with noisy labels from the perspective of self-supervised learning.

Original: We summarize our main contributions below:
Revised: We summarize our main contributions below:

Original: • We theoretically and experimentally show that by using self-supervised features to ﬁne-tune the network on noisy datasets, Cross Entropy itself is robust to label noise (Theorem 1–2).
Revised: • We theoretically and experimentally show that by using self-supervised features to ﬁne-tune the network on noisy datasets, Cross Entropy itself is robust to label noise (Theorem 1–2).

Original: The theory also answers the question of whether or not to ﬁx the encoder when performing ﬁne-tuning.
Revised: The theory also answers the question of whether or not to ﬁx the encoder when performing ﬁne-tuning.

Original: • We theoretically and experimentally show that by using self-supervised features, a regularizer commonly used in knowledge distillation (Hinton et al., 2015) (where the dataset does not contain label noise) can greatly alleviate over-ﬁtting problem of DNN on noisy datasets (Theorem
Revised: • We theoretically and experimentally show that by using self-supervised features, a regularizer commonly used in knowledge distillation (Hinton et al., 2015) (where the dataset does not contain label noise) can greatly alleviate over-ﬁtting problem of DNN on noisy datasets (Theorem

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Learning with Noisy Labels: Due to the over-ﬁtting problem of DNN, many works design robust loss to improve the robustness of neural networks.
Revised: Learning with Noisy Labels: Due to the over-ﬁtting problem of DNN, many works design robust loss to improve the robustness of neural networks.

Original: (Ghosh et al., 2017) proves MAE is inherently robust to label noise.
Revised: (Ghosh et al., 2017) proves MAE is inherently robust to label noise.

Original: However, MAE has a severe under-ﬁtting problem.
Revised: However, MAE has a severe under-ﬁtting problem.

Original: (Zhang & Sabuncu, 2018) propose a loss which can combine both the advantage of MAE and CE, exhibiting good performance on noisy datasets.
Revised: (Zhang & Sabuncu, 2018) propose a loss which can combine both the advantage of MAE and CE, exhibiting good performance on noisy datasets.

Original: (Liu & Guo, 2020) introduces peer loss, which is proven statistically robust to label noise without knowing noise rate.
Revised: (Liu & Guo, 2020) introduces peer loss, which is proven statistically robust to label noise without knowing noise rate.

Original: The extension of peer loss also shows good performance on instance-dependent label noise (Cheng et al., 2021; Zhu et al., 2021a).
Revised: The extension of peer loss also shows good performance on instance-dependent label noise (Cheng et al., 2021; Zhu et al., 2021a).

Original: Another efﬁcient approach to combat label noise is by sample selection (Jiang et al., 2018; Han et al., 2018;
Revised: Another efﬁcient approach to combat label noise is by sample selection (Jiang et al., 2018; Han et al., 2018;

Original: Yu et al., 2019;
Revised: Yu et al., 2019;

Original: Northcutt et al., 2021; Yao et al., 2020; Wei et al., 2020; Zhang et al., 2020).
Revised: Northcutt et al., 2021; Yao et al., 2020; Wei et al., 2020; Zhang et al., 2020).

Original: These methods regard “small loss” examples as clean ones and always involve training multiple networks to select clean samples.
Revised: These methods regard “small loss” examples as clean ones and always involve training multiple networks to select clean samples.

Original: Semi-supervised learning is also popular and effective on learning with noisy labels in recent years.
Revised: Semi-supervised learning is also popular and effective on learning with noisy labels in recent years.

Original: Some works (Li et al., 2020; Nguyen et al., 2020)
Revised: Some works (Li et al., 2020; Nguyen et al., 2020)

Original: ﬁrst perform clustering on the sample loss and divide the samples into clean ones and noisy ones.
Revised: ﬁrst perform clustering on the sample loss and divide the samples into clean ones and noisy ones.

Original: Then drop the labels of the ”noisy samples” and perform semi-supervised learning on all the samples.
Revised: Then drop the labels of the ”noisy samples” and perform semi-supervised learning on all the samples.

Original: Self-Supervised Learning:
Revised: Self-Supervised Learning:

Original: The goal of self-supervised learning (SSL) is to learn good presentation without using the information of the labels.
Revised: The goal of self-supervised learning (SSL) is to learn good presentation without using the information of the labels.

Original: Generally, the methods of SSL can be divided into two categories: designing pretext tasks or designing loss functions.
Revised: Generally, the methods of SSL can be divided into two categories: designing pretext tasks or designing loss functions.

Original: The designed tasks or losses do not involve any labels.
Revised: The designed tasks or losses do not involve any labels.

Original: Some popular tasks include patch orderings (Doersch et al., 2015; Noroozi &
Revised: Some popular tasks include patch orderings (Doersch et al., 2015; Noroozi &

Original: Favaro, 2016), tracking (Wang & Gupta, 2015) or clustering features (Caron et al., 2018; 2019).
Revised: Favaro, 2016), tracking (Wang & Gupta, 2015) or clustering features (Caron et al., 2018; 2019).

Original: However, the SSL performance of pretext tasks is limited.
Revised: However, the SSL performance of pretext tasks is limited.

Original: Recent SOTA methods for SSL is by designing contrastive loss functions.
Revised: Recent SOTA methods for SSL is by designing contrastive loss functions.

Original: The representative works include Moco (He et al., 2020) and
Revised: The representative works include Moco (He et al., 2020) and

Original: SimCLR (Chen et al., 2020) which train neural networks based on InfoNCE loss (Oord et al., 2018).
Revised: SimCLR (Chen et al., 2020) which train neural networks based on InfoNCE loss (Oord et al., 2018).

Original: In our paper, we also adopt InfoNCE for performing self-supervised training to get SSL pre-trained features.
Revised: In our paper, we also adopt InfoNCE for performing self-supervised training to get SSL pre-trained features.

Original: The ﬁrst part of our paper relates to the works that apply SSL features to perform ﬁnetuning on noisy dataset (Nodet et al., 2021; Ghosh & Lan, 2021) and our goal is to build theoretical understanding on this aspect.
Revised: The ﬁrst part of our paper relates to the works that apply SSL features to perform ﬁnetuning on noisy dataset (Nodet et al., 2021; Ghosh & Lan, 2021) and our goal is to build theoretical understanding on this aspect.

Original: Knowledge Distillation: The second part of our paper is very related to the research ﬁeld of knowledge distillation (KD).
Revised: Knowledge Distillation: The second part of our paper is very related to the research ﬁeld of knowledge distillation (KD).

Original: The original idea of KD can be traced back to model compression (Bucilu ˇ a et al., 2006), where authors demonstrate the knowledge acquired by a large ensemble of models can be transferred to a single small model.
Revised: The original idea of KD can be traced back to model compression (Bucilu ˇ a et al., 2006), where authors demonstrate the knowledge acquired by a large ensemble of models can be transferred to a single small model.

Original: (Hinton et al., 2015) generalize this idea to neural networks and show a small, shallow network can be improved through a teacher-student framework.
Revised: (Hinton et al., 2015) generalize this idea to neural networks and show a small, shallow network can be improved through a teacher-student framework.

Original: Due to its great applicability, KD has gained more and more attention in recent years and numerous methods have been proposed to perform efﬁcient distillation (Mirzadeh et al., 2020; Zhang et al., 2018; 2019).
Revised: Due to its great applicability, KD has gained more and more attention in recent years and numerous methods have been proposed to perform efﬁcient distillation (Mirzadeh et al., 2020; Zhang et al., 2018; 2019).

Original: However, the dataset used in KD is assumed to be clean.
Revised: However, the dataset used in KD is assumed to be clean.

Original: Thus it is hard to connect KD with learning with noisy labels.
Revised: Thus it is hard to connect KD with learning with noisy labels.

Original: In this paper, we theoretically and experimentally show that a regularizer generally used in KD (Park et al., 2019) can alleviate the over-ﬁtting problem on noisy data by using
Revised: In this paper, we theoretically and experimentally show that a regularizer generally used in KD (Park et al., 2019) can alleviate the over-ﬁtting problem on noisy data by using

Original: SSL features which offers a new alternative for dealing with label noise.
Revised: SSL features which offers a new alternative for dealing with label noise.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: We introduce preliminaries and notations including deﬁnitions and problem formulation.
Revised: We introduce preliminaries and notations including deﬁnitions and problem formulation.

Original: Problem Formulation: Consider a classiﬁcation problem on a set of N training examples denoted by D := { ( x n , y n ) } n ∈ [ N ] , where [ N ] := { 1 , 2 , · · · , N } is the set of example indices.
Revised: Problem Formulation: Consider a classiﬁcation problem on a set of N training examples denoted by D := { ( x n , y n ) } n ∈ [ N ] , where [ N ] := { 1 , 2 , · · · , N } is the set of example indices.

Original: Examples ( x n , y n ) are drawn according to random variables ( X, Y ) from a joint distribution D .
Revised: Examples ( x n , y n ) are drawn according to random variables ( X, Y ) from a joint distribution D .

Original: The classiﬁcation task aims to identify a classiﬁer C that maps X to Y accurately.
Revised: The classiﬁcation task aims to identify a classiﬁer C that maps X to Y accurately.

Original: Our theoretical analyses focus on binary classiﬁcations thus Y ∈ { 0 , 1 } .
Revised: Our theoretical analyses focus on binary classiﬁcations thus Y ∈ { 0 , 1 } .

Original: In real-world applications, the learner can only observe noisy labels.
Revised: In real-world applications, the learner can only observe noisy labels.

Original: For instance, human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly.
Revised: For instance, human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly.

Original: The label noise of each instance is assumed to be class-dependent (Liu & Tao, 2015), i.e., P ( (cid:101) Y | Y ) =
Revised: The label noise of each instance is assumed to be class-dependent (Liu & Tao, 2015), i.e., P ( (cid:101) Y | Y ) =

Original: P ( (cid:101) Y
Revised: P ( (cid:101) Y

Original: | X, Y ) .
Revised: | X, Y ) .

Original: Thus the error rates are deﬁned as e + = P ( (cid:101) Y = 

Revised: Thus the error rates are deﬁned as e + = P ( (cid:101) Y =

Original: | Y = 1) , e − =
Revised: | Y = 1) , e − =

Original: P ( (cid:101) Y = 1 | Y = 0) .
Revised: P ( (cid:101) Y = 1 | Y = 0) .

Original: The corresponding
Revised: The corresponding

Original: [Figure]
Revised: [Figure]

Original: noisy dataset and distribution are denoted by (cid:101) D :
Revised: noisy dataset and distribution are denoted by (cid:101) D :

Original: = { ( x n , ˜ y n ) } n ∈ [ N ] and (cid:101) D .
Revised: = { ( x n , ˜ y n ) } n ∈ [ N ] and (cid:101) D .

Original: Deﬁne the expected risk of a classiﬁer C as R ( C ) =
Revised: Deﬁne the expected risk of a classiﬁer C as R ( C ) =

Original: E D [ 1 ( C ( X ) (cid:54) =
Revised: E D [ 1 ( C ( X ) (cid:54) =

Original: Y )] .
Revised: Y )] .

Original: The goal is to learn a classiﬁer C from the noisy distribution (cid:101) D which also minimizes R ( C ) , i.e., learn the Bayes optimal classiﬁer such that
Revised: The goal is to learn a classiﬁer C from the noisy distribution (cid:101) D which also minimizes R ( C ) , i.e., learn the Bayes optimal classiﬁer such that

Original: C Bayes ( x ) = arg max i ∈{ 0 , 1 } P ( Y = i | X = x ) .
Revised: C Bayes ( x ) = arg max i ∈{ 0 , 1 } P ( Y = i | X = x ) .

Original: For better presentation, we deﬁne the following notations: X + = X
Revised: For better presentation, we deﬁne the following notations: X + = X

Original: | Y = 1 , X − =
Revised: | Y = 1 , X − =

Original: X | Y = 0 , and X clean = X
Revised: X | Y = 0 , and X clean = X

Original: | Y =
Revised: | Y =

Original: (cid:101) Y , X noisy =
Revised: (cid:101) Y , X noisy =

Original: X | Y (cid:54) = (cid:101) Y .
Revised: X | Y (cid:54) = (cid:101) Y .

Original: Evaluation of SSL (Self-Supervised Learning): SSL is usually evaluated by two steps: First, use
Revised: Evaluation of SSL (Self-Supervised Learning): SSL is usually evaluated by two steps: First, use

Original: SSL to train an encoder f with only unlabeled data X , then add a linear classiﬁer g following the pre-trained encoder f
Revised: SSL to train an encoder f with only unlabeled data X , then add a linear classiﬁer g following the pre-trained encoder f

Original: and only ﬁne-tune g on ( X, Y ) with a ﬁxed f .
Revised: and only ﬁne-tune g on ( X, Y ) with a ﬁxed f .

Original: The high-level intuition is that, if the encoder f is well learned by SSL, only ﬁne-tuning linear classiﬁer g is often sufﬁcient to achieve good performance on test data.
Revised: The high-level intuition is that, if the encoder f is well learned by SSL, only ﬁne-tuning linear classiﬁer g is often sufﬁcient to achieve good performance on test data.

Original: If the test performance is comparable to SL (Supervised
Revised: If the test performance is comparable to SL (Supervised

Original: Learning), we call the gap between SSL and SL is small (Chen et al., 2020).
Revised: Learning), we call the gap between SSL and SL is small (Chen et al., 2020).

Original: Denote by G the space of linear classiﬁer g .
Revised: Denote by G the space of linear classiﬁer g .

Original: Fine-tuning linear layer g on ( X, Y ) ∼ D can be represented as:
Revised: Fine-tuning linear layer g on ( X, Y ) ∼ D can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where CE denotes the Cross-Entropy loss.
Revised: where CE denotes the Cross-Entropy loss.

Original: Note the dimension of f ( X ) is determined by the network structure, e.g., 512 for ResNet34.
Revised: Note the dimension of f ( X ) is determined by the network structure, e.g., 512 for ResNet34.

Original: In binary classiﬁcations, g ( f ( X )) ∈ [0 , 1] where g ( f ( X )) < 0 .
Revised: In binary classiﬁcations, g ( f ( X )) ∈ [0 , 1] where g ( f ( X )) < 0 .

Original: 5 indicates predicting class- 0 and g ( f ( X )) >
Revised: 5 indicates predicting class- 0 and g ( f ( X )) >

Original: 5 corresponds to class- 1 .
Revised: 5 corresponds to class- 1 .

Original: g ( f ( X )) is supposed to predict the same label as C Bayes ( x ) .
Revised: g ( f ( X )) is supposed to predict the same label as C Bayes ( x ) .

Original:  R OBUSTNESS OF C ROSS -E NTROPY WITH SSL F EATURES
Revised:  R OBUSTNESS OF C ROSS -E NTROPY WITH SSL F EATURES

Original: We will analyze the robustness of Cross-Entropy with SSL features by comparing three different learning paths as illustrated in Figure 1.
Revised: We will analyze the robustness of Cross-Entropy with SSL features by comparing three different learning paths as illustrated in Figure 1.

Original: Path-1 is the traditional learning path that learns both encoder f and linear classiﬁer g at the same time.
Revised: Path-1 is the traditional learning path that learns both encoder f and linear classiﬁer g at the same time.

Original: Path-2 is the strategy applied in (Ghosh & Lan, 2021) that ﬁrstly pre-trains encoder f with SSL, then treats the pre-trained model as a network initialization and jointly ﬁne-tunes f and g .
Revised: Path-2 is the strategy applied in (Ghosh & Lan, 2021) that ﬁrstly pre-trains encoder f with SSL, then treats the pre-trained model as a network initialization and jointly ﬁne-tunes f and g .

Original: Path-3 is an alternate SSL-based path that ﬁrst learns the encoder f
Revised: Path-3 is an alternate SSL-based path that ﬁrst learns the encoder f

Original: then only ﬁne-tunes the linear classiﬁer g with ﬁxed f .
Revised: then only ﬁne-tunes the linear classiﬁer g with ﬁxed f .

Original: 3. T HEORETICAL T OOLS
Revised: 3. T HEORETICAL T OOLS

Original: We prepare some theoretical tools for our analyses.
Revised: We prepare some theoretical tools for our analyses.

Original: Our ﬁrst theorem focuses on demonstrating the effectiveness of only ﬁne-tuning linear classiﬁer g as in Path-3.
Revised: Our ﬁrst theorem focuses on demonstrating the effectiveness of only ﬁne-tuning linear classiﬁer g as in Path-3.

Original: We present Theorem 1 below.
Revised: We present Theorem 1 below.

Original: Theorem 1 Let g 1 = arg min g ∈G E D [ CE ( g ( f ( X )) , Y )] , g 2 = arg min g ∈G E (cid:101)
Revised: Theorem 1 Let g 1 = arg min g ∈G E D [ CE ( g ( f ( X )) , Y )] , g 2 = arg min g ∈G E (cid:101)

Original: D
Revised: D

Original: [ CE ( g ( f ( X )) , (cid:101) Y )] .
Revised: [ CE ( g ( f ( X )) , (cid:101) Y )] .

Original: Then if e + = e − < 0 .
Revised: Then if e + = e − < 0 .

Original: 5 , we have:
Revised: 5 , we have:

Original: [Equation]
Revised: [Equation]

Original: where f is ﬁxed encoder and g is the linear classiﬁer, g ( f ( · )) denotes the output whose value ranges from 0 to 1.
Revised: where f is ﬁxed encoder and g is the linear classiﬁer, g ( f ( · )) denotes the output whose value ranges from 0 to 1.

Original: Round ( p ) is a predictor function
Revised: Round ( p ) is a predictor function

Original: that outputs 1 if p > 0 .
Revised: that outputs 1 if p > 0 .

Original: 5 and outputs 0 otherwise.
Revised: 5 and outputs 0 otherwise.

Original: Theorem 1 shows with balanced error rates, simply ﬁne-tuning a linear classiﬁer g on the noisy data distribution (cid:101) D can achieve the same decision boundary as the optimal linear classiﬁer obtained from the corresponding clean distribution D .
Revised: Theorem 1 shows with balanced error rates, simply ﬁne-tuning a linear classiﬁer g on the noisy data distribution (cid:101) D can achieve the same decision boundary as the optimal linear classiﬁer obtained from the corresponding clean distribution D .

Original: i.e., g 1 ( f ) and g 2 ( f ) have the same predictions for all the samples.
Revised: i.e., g 1 ( f ) and g 2 ( f ) have the same predictions for all the samples.

Original: Theorem 1 can be generalized to the case with an arbitrary classiﬁer beyond linear.
Revised: Theorem 1 can be generalized to the case with an arbitrary classiﬁer beyond linear.

Original: [Figure]
Revised: [Figure]

Original: However, admittedly with limited data, training complicated classiﬁers is hard to converge to the optimal decision boundary.
Revised: However, admittedly with limited data, training complicated classiﬁers is hard to converge to the optimal decision boundary.

Original: We defer more details to the next subsection.
Revised: We defer more details to the next subsection.

Original: We then evaluate the former part of Path-3, i.e., the performance of SSL.
Revised: We then evaluate the former part of Path-3, i.e., the performance of SSL.

Original: Recall in Section 2, SSL is usually evaluated by performance gap between f ◦ g Bayes and C Bayes , where g Bayes is the optimal linear classiﬁer trained on D , f ◦ g Bayes denotes the joint model given by g Bayes ( f ( X )) .
Revised: Recall in Section 2, SSL is usually evaluated by performance gap between f ◦ g Bayes and C Bayes , where g Bayes is the optimal linear classiﬁer trained on D , f ◦ g Bayes denotes the joint model given by g Bayes ( f ( X )) .

Original: We consider a tractable case in Assumption 1.
Revised: We consider a tractable case in Assumption 1.

Original: Assumption 1 The encoder outputs f
Revised: Assumption 1 The encoder outputs f

Original: ( X + ) and f ( X − ) follow Gaussian distribution with parameters ( µ 1 , Σ) and ( µ 2 , Σ) , where Σ = σ 2 · I , I is the identity matrix.
Revised: ( X + ) and f ( X − ) follow Gaussian distribution with parameters ( µ 1 , Σ) and ( µ 2 , Σ) , where Σ = σ 2 · I , I is the identity matrix.

Original: Assumption 1 states that the self-supervised features for each class follow simple Gaussian distributions.
Revised: Assumption 1 states that the self-supervised features for each class follow simple Gaussian distributions.

Original: We check the effectiveness of this assumption by Figure 2.
Revised: We check the effectiveness of this assumption by Figure 2.

Original: It can be observed that the features of each class may have overlaps, but a good SSL method is supposed to return features with good separations (small overlaps).
Revised: It can be observed that the features of each class may have overlaps, but a good SSL method is supposed to return features with good separations (small overlaps).

Original: In Assumption 1, we use || µ 1 − µ 2 || and σ to capture the overlapping area of two classes.
Revised: In Assumption 1, we use || µ 1 − µ 2 || and σ to capture the overlapping area of two classes.

Original: If || µ 1 − µ 2 || is large and σ is small, then there exists small overlapping.
Revised: If || µ 1 − µ 2 || is large and σ is small, then there exists small overlapping.

Original: Based on this assumption, we show the performance of SSL in Theorem 2.
Revised: Based on this assumption, we show the performance of SSL in Theorem 2.

Original: Theorem 

Revised: Theorem

Original: If P ( Y = 1) = P ( Y = 0) , the risk (error rate) of Bayes optimal classiﬁer f ◦ g Bayes follows as:
Revised: If P ( Y = 1) = P ( Y = 0) , the risk (error rate) of Bayes optimal classiﬁer f ◦ g Bayes follows as:

Original: [Equation]
Revised: [Equation]

Original: where Φ is the cumulative distribution function (CDF) of the standard Gaussian distribution.
Revised: where Φ is the cumulative distribution function (CDF) of the standard Gaussian distribution.

Original: Wrap-up With Theorem 1, we know CE is robust, the performance of which is subject to f .
Revised: Wrap-up With Theorem 1, we know CE is robust, the performance of which is subject to f .

Original: Theorem 2 implies that if SSL features learned by f exhibit good property, i.e., when || µ
Revised: Theorem 2 implies that if SSL features learned by f exhibit good property, i.e., when || µ

Original: 1 − µ 2 || is large and σ is small, only ﬁne-tuning g can approach the Bayes optimal classiﬁer C Bayes .
Revised: 1 − µ 2 || is large and σ is small, only ﬁne-tuning g can approach the Bayes optimal classiﬁer C Bayes .

Original: Therefore, good SSL features induce high performance.
Revised: Therefore, good SSL features induce high performance.

Original: In summary, Theorem 1 and Theorem 2 connect SSL features with robustness and generalization ability of CE loss, providing an insight on why SSL features improve classiﬁcation with label noise.
Revised: In summary, Theorem 1 and Theorem 2 connect SSL features with robustness and generalization ability of CE loss, providing an insight on why SSL features improve classiﬁcation with label noise.

Original: 3. C AN W E F IX THE E NCODER ?
Revised: 3. C AN W E F IX THE E NCODER ?

Original: We compare the performance of Path-2 with Path-3 in this subsection.
Revised: We compare the performance of Path-2 with Path-3 in this subsection.

Original: 3.2. T HEORETICAL A NALYSES
Revised: 3.2. T HEORETICAL A NALYSES

Original: Denote by VC ( G ) the VC-dimension of G .
Revised: Denote by VC ( G ) the VC-dimension of G .

Original: Always ﬁx good encoders When we ﬁne-tune g on ﬁxed f , the learning errors will come from two parts: the encoder error and the classiﬁcation error.
Revised: Always ﬁx good encoders When we ﬁne-tune g on ﬁxed f , the learning errors will come from two parts: the encoder error and the classiﬁcation error.

Original: The encoder error is bounded in Theorem 2.
Revised: The encoder error is bounded in Theorem 2.

Original: Noting that g ∈ G is a linear classiﬁer with relatively low VC-dimension and CE is a calibrated and convex function, the optimization can achieve global optimum with enough samples (Bartlett et al., 2006), e.g., w.p. 1 − δ , the learning error will be ε with Θ( VC ( G )
Revised: Noting that g ∈ G is a linear classiﬁer with relatively low VC-dimension and CE is a calibrated and convex function, the optimization can achieve global optimum with enough samples (Bartlett et al., 2006), e.g., w.p. 1 − δ , the learning error will be ε with Θ( VC ( G )

Original: +ln1 /δ ε ) i.i.d.
Revised: +ln1 /δ ε ) i.i.d.

Original: instances (Vapnik,
Revised: instances (Vapnik,

Original: [Figure]
Revised: [Figure]

Revised: Detailed experimental setting are reported in the Appendix.

Original: Recall Theorem 1 also guarantees the optimality of g learned under the noisy distribution (cid:101) D .
Revised: Recall Theorem 1 also guarantees the optimality of g learned under the noisy distribution (cid:101) D .

Original: Therefore, given a good encoder with large enough (cid:107) µ 1 − µ 2 (cid:107) σ, we should keep it ﬁxed since the simple linear classiﬁer g trained under label noise only requires N = Θ( VC ( G )
Revised: Therefore, given a good encoder with large enough (cid:107) µ 1 − µ 2 (cid:107) σ, we should keep it ﬁxed since the simple linear classiﬁer g trained under label noise only requires N = Θ( VC ( G )

Original: +
Revised: +

Original: ln1 /δ ε ) i.i.d.
Revised: ln1 /δ ε ) i.i.d.

Original: instances in (cid:101) D to achieve around ε errors with high probability, which performs well enough to approximate C Bayes .
Revised: instances in (cid:101) D to achieve around ε errors with high probability, which performs well enough to approximate C Bayes .

Original: Updating f is not stable If f is unﬁxed, due to the non-convexity and over-ﬁtting capability of f , network trained on vanilla (cid:101) D will likely and ultimately overﬁt to noisy labels (Zhang et al., 2016), particularly when the number of instances is insufﬁcient.
Revised: Updating f is not stable If f is unﬁxed, due to the non-convexity and over-ﬁtting capability of f , network trained on vanilla (cid:101) D will likely and ultimately overﬁt to noisy labels (Zhang et al., 2016), particularly when the number of instances is insufﬁcient.

Original: For example, w.p. 1 − δ , jointly learning f and g and achieving error ε requires Θ( VC ( F◦G )
Revised: For example, w.p. 1 − δ , jointly learning f and g and achieving error ε requires Θ( VC ( F◦G )

Original: +
Revised: +

Original: ln1 /δ ε ) i.i.d.
Revised: ln1 /δ ε ) i.i.d.

Original: instances, which is far more than the number required in only tuning g due to the huge capacity of DNNs.
Revised: instances, which is far more than the number required in only tuning g due to the huge capacity of DNNs.

Original: Note the effective i.i.d.
Revised: Note the effective i.i.d.

Original: instances would decrease with the increase of noise rates.
Revised: instances would decrease with the increase of noise rates.

Original: Although a good initialization contributes to better convergence, f will be misled by label noise in extreme cases.
Revised: Although a good initialization contributes to better convergence, f will be misled by label noise in extreme cases.

Original: Therefore it is unreasonable to assume that CE will stably induce the network to approximate the Bayes optimal classiﬁer under label noise.
Revised: Therefore it is unreasonable to assume that CE will stably induce the network to approximate the Bayes optimal classiﬁer under label noise.

Original: In other words, if f is not ﬁxed, due to the overﬁtting capability of f , the outputs of f ( X ) would be contaminated by the label noise before g .
Revised: In other words, if f is not ﬁxed, due to the overﬁtting capability of f , the outputs of f ( X ) would be contaminated by the label noise before g .

Original: In contrast, the original SSL features are independent from the label noise if f is ﬁxed.
Revised: In contrast, the original SSL features are independent from the label noise if f is ﬁxed.

Original: We therefore conclude that CE on vanilla (cid:101) D is not robust.
Revised: We therefore conclude that CE on vanilla (cid:101) D is not robust.

Original: Note Nodet et al.
Revised: Note Nodet et al.

Original: (2021) claimed the encoder should not be ﬁxed during ﬁne-tuning based on some experiments.
Revised: (2021) claimed the encoder should not be ﬁxed during ﬁne-tuning based on some experiments.

Original: We include more discussions in Appendix.
Revised: We include more discussions in Appendix.

Original: 3.2. E MPIRICAL EVIDENCE
Revised: 3.2. E MPIRICAL EVIDENCE

Original: We conduct experiments in Figure 3 to support our theorems and analyses above.
Revised: We conduct experiments in Figure 3 to support our theorems and analyses above.

Original: Overall performance From Figure 3(a)–(c), when encoder f is ﬁxed, the performance of CE (ﬁxed encoder) exhibits great robustness as noise rate increases and only has a little drop when the symmetric noise rate approximates the maximum theoretical value for each dataset (0.5 for DogCat, 0.9 for CIFAR10) which veriﬁes Theorem 1 and Theorem 2.
Revised: Overall performance From Figure 3(a)–(c), when encoder f is ﬁxed, the performance of CE (ﬁxed encoder) exhibits great robustness as noise rate increases and only has a little drop when the symmetric noise rate approximates the maximum theoretical value for each dataset (0.5 for DogCat, 0.9 for CIFAR10) which veriﬁes Theorem 1 and Theorem 2.

Original: Note that the drop is because, to reach the condition speciﬁed in Equation (1), a substantial number of training samples are needed.
Revised: Note that the drop is because, to reach the condition speciﬁed in Equation (1), a substantial number of training samples are needed.

Original: However, for example, each class only has 500 samples in CIFAR100.
Revised: However, for example, each class only has 500 samples in CIFAR100.

Original: As the noise rate increases, the number of clean samples will further decrease which makes the classiﬁer hard to learn from the distribution.
Revised: As the noise rate increases, the number of clean samples will further decrease which makes the classiﬁer hard to learn from the distribution.

Original: Fixed encoder beneﬁts extreme label noise Next, we observe the performance gap between SSL and SL when datasets are clean.
Revised: Fixed encoder beneﬁts extreme label noise Next, we observe the performance gap between SSL and SL when datasets are clean.

Original: This gap can be visualized in Figure 3 (a) (b) (c) at noise rate = 0 .
Revised: This gap can be visualized in Figure 3 (a) (b) (c) at noise rate = 0 .

Original: For DogCat , the gap is very small, thus CE with ﬁxed encoder always exhibits better performance.
Revised: For DogCat , the gap is very small, thus CE with ﬁxed encoder always exhibits better performance.

Original: For CIFAR10, the gap is moderate, the performances of CE (ﬁxed encoder) and CE (unﬁxed encoder) are close when noise rate is small.
Revised: For CIFAR10, the gap is moderate, the performances of CE (ﬁxed encoder) and CE (unﬁxed encoder) are close when noise rate is small.

Original: However, when noise rate is high, CE with ﬁxed encoder still exhibits better performance.
Revised: However, when noise rate is high, CE with ﬁxed encoder still exhibits better performance.

Original: For CIFAR100, the gap is large, thus CE with ﬁxed encoder shows better performance only when noise rate is over 0.7.
Revised: For CIFAR100, the gap is large, thus CE with ﬁxed encoder shows better performance only when noise rate is over 0.7.

Original: Fixed encoder has better convergence Note that the performance in Figure 3 (a) (b) (c) are selected from best epoch accuracy.
Revised: Fixed encoder has better convergence Note that the performance in Figure 3 (a) (b) (c) are selected from best epoch accuracy.

Original: If we take the last epoch accuracy into consideration.
Revised: If we take the last epoch accuracy into consideration.

Original: CE with ﬁxed[Table] encoder has great advantage over unﬁxed encoder.
Revised: CE with ﬁxed encoder has great advantage over unﬁxed encoder.

Original: Figure 3 (d) depicts the performance of CE in  terms of training epochs under 0.6 symmetric noise rate on CIFAR10.
Revised: Figure 3 (d) depicts the performance of CE in [Table] terms of training epochs under 0.6 symmetric noise rate on CIFAR10.

Original: We ﬁnd that even though these two approaches have almost same best epoch accuracy, CE with ﬁxed encoder is more robust as training proceeds.
Revised: We ﬁnd that even though these two approaches have almost same best epoch accuracy, CE with ﬁxed encoder is more robust as training proceeds.

Original: For CE with unﬁxed encoder, since the self-supervised pre-trained network’s weights serve as a better initialization, it shows good performance in the beginning.
Revised: For CE with unﬁxed encoder, since the self-supervised pre-trained network’s weights serve as a better initialization, it shows good performance in the beginning.

Original: However, because of the memorizing effect of DNN, it will still over-ﬁt to noisy labels in the end.
Revised: However, because of the memorizing effect of DNN, it will still over-ﬁt to noisy labels in the end.

Original: Generalize to class- and instance-dependent label noise: Theorem 1 shows that CE is robust to label noise when e − =
Revised: Generalize to class- and instance-dependent label noise: Theorem 1 shows that CE is robust to label noise when e − =

Original: e + .
Revised: e + .

Original: However, for a more general label noise such as asymmetric classdependent label noise or instance-dependent label noise, CE with ﬁxed encoder on vanilla ( f ( X ) , (cid:101) Y ) is hard to be guaranteed to be robust because for instance noise, noise rate varies for each class.
Revised: However, for a more general label noise such as asymmetric classdependent label noise or instance-dependent label noise, CE with ﬁxed encoder on vanilla ( f ( X ) , (cid:101) Y ) is hard to be guaranteed to be robust because for instance noise, noise rate varies for each class.

Original: But it does not mean SSL features can not help.
Revised: But it does not mean SSL features can not help.

Original: We may still ﬁx the encoder and perform a simple down-sampling to balance the noisy dataset before ﬁne-tuning the linear classiﬁer.
Revised: We may still ﬁx the encoder and perform a simple downsampling to balance the noisy dataset before ﬁne-tuning the linear classiﬁer.

Original: For example, for class-dependent label noise that is not symmetric, the down-sampling is conducted to make P ( (cid:101) Y = i ) =
Revised: The down-sampling is conducted to make P ( (cid:101) Y = i ) =

Original: P ( (cid:101) Y = j ) in the noisy dataset which can well approximate the condition in Theorem(Illustration is deferred to Appendix).
Revised: P ( (cid:101) Y = j ) in the noisy dataset which can well approximate the condition in Theorem 1 (Illustration is deferred to Appendix).

Original: Although not theoretically guaranteed, similar down-sampling strategy should also beneﬁt the learning with instance-dependent label noise, which may be highly unbalanced.
Revised: Note that with down-sampling, the target distribution of clean labels also changes. However, Theorem 1 is proved without specifying the distribution of clean labels, suggesting that classiﬁer is consistent even for imbalanced dataset under symmetric label noise. Secondly, with SSL features, classiﬁer is robust to imbalanced imbalanced dataset (Yang & Xu, 2020). Thus down-sampling strategy to reduce error rate imbalances is helpful for other noise types beyond symmetric.

Original: We perform an experiment in Table 1 to verify the effectiveness of the down-sampling in the setting of instance-dependent label noise (generation of label noise is followed by (Cheng et al., 2021)).
Revised: We perform an experiment in Table 1 to verify the effectiveness of the down-sampling in the setting of instance-dependent label noise (generation of label noise is followed by (Cheng et al., 2021)).

Original: From the experiments, if no sampling strategy is conducted, CE with ﬁxed encoder does not show many beneﬁts.
Revised: From the experiments, if no sampling strategy is conducted, CE with ﬁxed encoder does not show many beneﬁts.

Original: In some cases, it even performs worse than CE baseline (unﬁxed f with random init).
Revised: In some cases, it even performs worse than CE baseline (unﬁxed f with random init).

Original: However, with down-sampling, CE with ﬁxed encoder even outperforms CORES (Cheng et al., 2021) by a large margin.
Revised: However, with down-sampling, CE with ﬁxed encoder even outperforms CORES (Cheng et al., 2021) by a large margin.

Original: Even though our main purpose in the paper is not to develop new methods for achieving SOTA, we think this down-sampling strategy motivated by Theoremcan inspire further research on instance-dependent label noise by using SSL features.
Revised: Even though our main purpose in the paper is not to develop new methods for achieving SOTA, we think this downsampling strategy motivated by Theorem 1 can inspire further research on instance-dependent label noise by using SSL features.

Original: 3. T AKEAWAYS
Revised: 3. T AKEAWAYS

Original: Our both theoretical and empirical results demonstrate that: We need to simply ﬁx the encoder when 1) the encoder is sufﬁciently good, 2) the high noise rate decreases the number of effective i.i.d.
Revised: Our both theoretical and empirical results demonstrate that: We need to simply ﬁx the encoder when 1) the encoder is sufﬁciently good, 2) the high noise rate decreases the number of effective i.i.d.

Original: instances to a certain level, 3) the over-parameterized model makes it easy to overﬁt label noise.
Revised: instances to a certain level, 3) the over-parameterized model makes it easy to overﬁt label noise.

Original: However, there are some concerns: 1) If the gap between SL and SSL is relatively large, CE with unﬁxed encoder has more advantage on small noise rate (e.g., Figure 3 (c)); 2) Consider an online classiﬁcation system which always needs new training data to update itself.
Revised: However, there are some concerns: 1) If the gap between SL and SSL is relatively large, CE with unﬁxed encoder has more advantage on small noise rate (e.g., Figure 3 (c)); 2) Consider an online classiﬁcation system which always needs new training data to update itself.

Original: If we only ﬁne-tune the linear classiﬁer for keeping robustness to label noise, then due to the limited learnability of afﬁne functions, this online classiﬁcation system may learn very limited knowledge from unseen data.
Revised: If we only ﬁne-tune the linear classiﬁer for keeping robustness to label noise, then due to the limited learnability of afﬁne functions, this online classiﬁcation system may learn very limited knowledge from unseen data.

Original: Therefore, it is necessary to update the encoder while keeping it stable and robust to label noise, which is the focus in the next section.
Revised: Therefore, it is necessary to update the encoder while keeping it stable and robust to label noise, which is the focus in the next section.

Original:  R EGULARIZER FROM KD IMPROVES NETWORK ROBUSTNESS
Revised:  R EGULARIZER FROM KD IMPROVES NETWORK ROBUSTNESS

Original: In this section, we show that a regularizer from knowledge distillation (KD) (Park et al., 2019) can well alleviate the over-ﬁtting problem by using the structure of SSL features.
Revised: In this section, we show that a regularizer from knowledge distillation (KD) (Park et al., 2019) can well alleviate the over-ﬁtting problem by using the structure of SSL features.

Original: Different from classical
Revised: Different from classical

Original: KD settings where the dataset is clean, our analyses aim at regularizing classiﬁers learned with noisy supervisions by SSL features.
Revised: KD settings where the dataset is clean, our analyses aim at regularizing classiﬁers learned with noisy [Figure]

Original: Understanding the working mechanism is important since it may contribute to the community a new perspective to learn with label noise by distilling information
Revised: 𝑓

Original: [Figure]
Revised: [Figure]

Original: from SSL features.
Revised: supervisions by SSL features.

Revised: Understanding the working mechanism is important since it may contribute to the community a new perspective to learn with label noise by distilling information from SSL features.

Original: In the rest of this section, we will ﬁrst provide theoretical understanding towards the regularizer under label noise, then perform experiments on different settings to test our analyses.
Revised: In the rest of this section, we will ﬁrst provide theoretical understanding towards the regularizer under label noise, then perform experiments on different settings to test our analyses.

Original: 4. T RAINING F RAMEWORK
Revised: 4. T RAINING F RAMEWORK

Original: The training framework is shown in Figure 4, where a new learning path (SSL training)
Revised: The training framework is shown in Figure 4, where a new learning path (SSL training)

Original: f → h is added to be parallel to Path-2 f → g (SL-training) in Figure 1.
Revised: f → h is added to be parallel to Path-2 f → g (SL-training) in Figure 1.

Original: The newly added projection head h is one-hidden-layer MLP (Multi Layer Perceptron) whose output represents SSL features (after dimension reduction).
Revised: The newly added projection head h is one-hidden-layer MLP (Multi Layer Perceptron) whose output represents SSL features (after dimension reduction).

Original: Its output is employed to regularize the output of linear classiﬁer g as
Revised: Its output is employed to regularize the output of linear classiﬁer g as

Original: Reg ( h ( f ( X )) , g ( f ( X ))) .
Revised: Reg ( h ( f ( X )) , g ( f ( X ))) .

Original: InfoNCE from SimCLR (Chen et al., 2020) is adopted for SSL training and CE is for SL training.
Revised: InfoNCE from SimCLR (Chen et al., 2020) is adopted for SSL training and CE is for SL training.

Original: InfoNCE and CE share a common encoder, inspired by the design of self distillation (Zhang et al., 2019).
Revised: InfoNCE and CE share a common encoder, inspired by the design of self distillation (Zhang et al., 2019).

Original: The loss function is deﬁned as:
Revised: The loss function is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, SL features is supposed to be beneﬁted from the structure information from SSL features, e.g., clusterability (Zhu et al., 2021b) that instances with similar SSL features should have the same true label and instance with different SSL features should have different true labels.
Revised: Intuitively, SL features is supposed to be beneﬁted from the structure information from SSL features, e.g., clusterability (Zhu et al., 2021b) that instances with similar SSL features should have the same true label and instance with different SSL features should have different true labels.

Original: Mathematically, let t i
Revised: Mathematically, let t i

Original: = h
Revised: = h

Original: ( f ( x i )) , s i
Revised: ( f ( x i )) , s i

Original: = g ( f ( x i )) and X N to be the set of N tuples of data samples.
Revised: = g ( f ( x i )) and X N to be the set of N tuples of data samples.

Original: The distance between t i and t j can be represented as φ w ( t i , t j ) =
Revised: The distance between t i and t j can be represented as φ w ( t i , t j ) =

Original: 1 m (cid:107) t i − t j (cid:107) w , where w ∈ { 1 , 2 } and m is a normalization term:
Revised: 1 m (cid:107) t i − t j (cid:107) w , where w ∈ { 1 , 2 } and m is a normalization term:

Original: [Equation]
Revised: [Equation]

Original: Then the expectation of Reg ( h ( X ) , g ( X )) in Equation (3) can be estimated by:
Revised: Then the expectation of Reg ( h ( X ) , g ( X )) in Equation (3) can be estimated by:

Original: [Equation]
Revised: [Equation]

Original: where d ( · ) is a distance measure for two inputs.
Revised: where d ( · ) is a distance measure for two inputs.

Original: Popular choices are l 1 , l 2 or square l 2 distance.
Revised: Popular choices are l 1 , l 2 or square l 2 distance.

Original: 4. A NALYTICAL F RAMEWORK
Revised: 4. A NALYTICAL F RAMEWORK

Original: Under the label noise setting, denote by d i,j = d ( φ w ( t i , t j ) , φ w ( s i , s j )) , we have the following decomposition: 
Revised: Under the label noise setting, denote by d i,j = d ( φ w ( t i , t j ) , φ w ( s i , s j )) , we have the following decomposition: | [Equation]

Revised: 2 | (cid:88) ( x i ,x j ) ∈Xd

Revised: i,j = |X 2 | (cid:16) (cid:88) ( x i

Original: [Equation]
Revised: [Equation]

Revised: x j ) ∈X 2 clean d i,j (cid:124) (cid:123)(cid:122) (cid:125)

Revised: Term-

Revised: + (cid:88) ( x i ,x j ) ∈X

Revised: [Equation]

Revised: noisy

Revised: [Equation]

Revised: (cid:124) (cid:123)(cid:122) (cid:125) Term

Revised: [Equation]

Revised: + (cid:88)

Revised: [Equation]

Revised: (cid:123)(cid:122) (cid:125)

Revised: [Equation]

Original: where X = X clean (cid:83)
Revised: where X = X clean (cid:83)

Original: X noisy , X noisy denotes the set of samples whose labels are ﬂipped.
Revised: X noisy , X noisy denotes the set of samples whose labels are ﬂipped.

Original: Note the regularizer mainly works when SSL features “disagree” with SL features, i.e., Term-3.
Revised: Note the regularizer mainly works when SSL features “disagree” with SL features, i.e., Term-3.

Original: For further analyses, we write Term-3 in the form of expectation with d chosen as square l 2 distance, i.e. , MSE loss:
Revised: For further analyses, we write Term-3 in the form of expectation with d chosen as square l 2 distance, i.e. , MSE loss:

Original: [Equation]
Revised: [Equation]

Original: where m 1 and m 2 are normalization terms in Equation (4).
Revised: where m 1 and m 2 are normalization terms in Equation (4).

Original: Note in L c , we use w = 1 for SL features and w = 2 for SSL features.
Revised: Note in L c , we use w = 1 for SL features and w = 2 for SSL features.

Original: 1 Denote the variance by var ( · ) .
Revised: 1 Denote the variance by var ( · ) .

Original: Deﬁne notations:
Revised: Deﬁne notations:

Original: X noisy + := X | ( (cid:101) Y = 1 , Y = 0) , X noisy − :=
Revised: [Equation]

Original: X | ( (cid:101) Y = 0 , Y = 1) .
Revised: [Equation]

Original: To theoretically measure and quantify how feature correction relates to network robustness, we make three assumptions as follows:
Revised: To theoretically measure and quantify how feature correction relates to network robustness, we make three assumptions as follows:

Original: Assumption 2 (Memorize clean instances) ∀ n [Equation] x n  ) [Equation]
Revised: Assumption 2 (Memorize clean instances) ∀ n ∈ { n | ˜ y n = y n } , CE ( g ( f ( x n )) , y n ) = 0 .

Original: , y [Equation] = 0 . Assumption 3 (Same overﬁtting) var ( g ( f ( X noisy + )))
Revised: Assumption 3 (Same overﬁtting) var ( g ( f ( X noisy + )))

Original: = 0 and var ( g ( f ( X noisy − )))
Revised: = 0 and var ( g ( f ( X noisy − )))

Original: = 0 .
Revised: = 0 .

Original: Assumption 4 (Gaussian-distributed SSL features)
Revised: Assumption 4 (Gaussian-distributed SSL features)

Original: The SSL features follow Gaussian distributions, i.e., h ( f ( X +1 ))
Revised: The SSL features follow Gaussian distributions, i.e., h ( f ( X +1 ))

Original: ∼ N ( µ 1 , Σ) and h ( f ( X − 1 ))
Revised: ∼ N ( µ 1 , Σ) and h ( f ( X − 1 ))

Original: ∼ N ( µ 2 , Σ) .
Revised: ∼ N ( µ 2 , Σ) .

Original: Assumption 2 implies that a DNN has conﬁdent predictions on clean samples.
Revised: Assumption 2 implies that a DNN has conﬁdent predictions on clean samples.

Original: Assumption 3 implies that a DNN has the same degree of overﬁtting for each noisy sample.
Revised: Assumption 3 implies that a DNN has the same degree of overﬁtting for each noisy sample.

Original: For example, an overparameterized DNN can memorize all the noisy labels (Zhang et al., 2016; Liu, 2021).
Revised: For example, an overparameterized DNN can memorize all the noisy labels (Zhang et al., 2016; Liu, 2021).

Original: Thus these two assumptions are reasonable.
Revised: Thus these two assumptions are reasonable.

Original: Assumption 4 follows Assumption 1.
Revised: Assumption 4 follows Assumption 1.

Original: Note in Figure 4, SSL features are from h ( f ( X )) rather than f ( X ) .
Revised: Note in Figure 4, SSL features are from h ( f ( X )) rather than f ( X ) .

Original: 4. T HEORETICAL U NDERSTANDING
Revised: 4. T HEORETICAL U NDERSTANDING

Original: Based on Assumptions 2–4, we present Theorem 3 to analyze the effect of L c .
Revised: Based on Assumptions 2–4, we present Theorem 3 to analyze the effect of L c .

Original: Recall X noisy
Revised: Recall X noisy

Original: + :=
Revised: + :=

Original: X |
Revised: X |

Original: ( (cid:101) Y = 1 , Y = 0) , X noisy − :=
Revised: ( (cid:101) Y = 1 , Y = 0) , X noisy − :=

Original: X | ( (cid:101)
Revised: X | ( (cid:101)

Original: Y = 0 , Y = 1) .
Revised: Y = 0 , Y = 1) .

Original: Theorem 3 When e − = e + and P ( Y = 1) =
Revised: Theorem 3 When e − = e + and P ( Y = 1) =

Original: P ( Y = 0) , minimizing L c respect to g ( f ( X noisy )) on
Revised: P ( Y = 0) , minimizing L c respect to g ( f ( X noisy )) on

Original: DNN results in the following solutions:
Revised: DNN results in the following solutions:

Original: [Equation]
Revised: [Equation]

Original:  where ∆(Σ , µ 1 , µ 2 ) :
Revised: , where ∆(Σ , µ 1 , µ 2 ) :

Original: = 8 · tr (Σ) / || µ 1 − µ 2 || 2 , tr ( · ) denotes the matrix trace, .
Revised: = 8 · tr (Σ) / || µ 1 − µ 2 || 2 , tr ( · ) denotes the matrix trace, .

Original: Theorem 3 reveals a clean relationship between the quality of SSL features (given by h ( f ( X )) ) and the network robustness on noisy samples.
Revised: Theorem 3 reveals a clean relationship between the quality of SSL features (given by h ( f ( X )) ) and the network robustness on noisy samples.

Original: Note the true label for X noisy + is 0 and the true label for X noisy − is 1 .
Revised: Note the true label for X noisy + is 0 and the true label for X noisy − is 1 .

Original: Theorem 3 shows the regularizer is leading the classiﬁer to predict the feature with corrupted label to its corresponding true label, and the guidance is particular strong when ∆(Σ , µ 1 , µ 2 ) → 0 , e.g., tr (Σ) → 0 or (cid:107) µ 1 − µ 2 (cid:107) → ∞ , such that E X noisy
Revised: Theorem 3 shows the regularizer is leading the classiﬁer to predict the feature with corrupted label to its corresponding true label, and the guidance is particular strong when ∆(Σ , µ 1 , µ 2 ) → 0 , e.g., tr (Σ) → 0 or (cid:107) µ 1 − µ 2 (cid:107) → ∞ , such that E X noisy

Original: + [ g ( f ( X noisy + ))]
Revised: + [ g ( f ( X noisy + ))]

Original: → 0 and
Revised: → 0 and

Original: E X noisy − [ g ( f ( X noisy − ))]
Revised: E X noisy − [ g ( f ( X noisy − ))]

Original: → 1 .
Revised: → 1 .

Original: Beyond this ideal case, we discuss more interesting ﬁndings as:
Revised: Beyond this ideal case, we discuss more interesting ﬁndings as:

Original: • Worst case: The regularizer can always have a positive effect to avoid memorizing wrong labels as long as the model f
Revised: • Worst case: The regularizer can always have a positive effect to avoid memorizing wrong labels as long as the model f

Original: ◦ h is better than randomly initialized models.
Revised: ◦ h is better than randomly initialized models.

Original: This is because ∆(Σ , µ 1 , µ 2 ) > 0 when µ 1 and µ 2 are different, and the regularizer is leading the classiﬁer to the true direction when ∆(Σ , µ 1 , µ 2 ) > 0 .
Revised: This is because ∆(Σ , µ 1 , µ 2 ) > 0 when µ 1 and µ 2 are different, and the regularizer is leading the classiﬁer to the true direction when ∆(Σ , µ 1 , µ 2 ) > 0 .

Original: • Practical case: The model f ◦ h is always better than randomly initialized models since SL training can guarantee the performance of encoder f on clean instances, which may generalize to noisy instances, and SSL training can further improves the quality of f ◦ h .
Revised: • Practical case: The model f ◦ h is always better than randomly initialized models since SL training can guarantee the performance of encoder f on clean instances, which may generalize to noisy instances, and SSL training can further improves the quality of f ◦ h .

Original: Thus in practical cases, using SSL features to regularize learning with noisy labels is always beneﬁcial.
Revised: Thus in practical cases, using SSL features to regularize learning with noisy labels is always beneﬁcial.

Original: • More possibilities: Note the proof of Theorem 3 does not rely on any SSL training process.
Revised: • More possibilities: Note the proof of Theorem 3 does not rely on any SSL training process.

Original: From our ﬁrst ﬁnding, we know any encoder helps if it is better than random models.
Revised: From our ﬁrst ﬁnding, we know any encoder helps if it is better than random models.

Original: This makes it possible to use some pre-trained encoders from other tasks.
Revised: This makes it possible to use some pre-trained encoders from other tasks.

Original: High-level understanding on structure regularization: Even though we have built Theorem 3 to show SL features can beneﬁt from the structure of SSL features by performing regularization, there  still lacks high-level understanding of what the regularization is exactly doing.
Revised: High-level understanding on structure regularization: Even though we have built Theorem 3 to show SL features can beneﬁt from the structure of SSL features by performing regularization, there [Figure] still lacks high-level understanding of what the regularization is exactly doing.

Original: Here we provide an insight in Theorem 4 which shows the regularization is implicitly maximizing mutual information between SL features and SSL features.
Revised: Here we provide an insight in Theorem 4 which shows the regularization is implicitly maximizing mutual information between SL features and SSL features.

Original: [Figure] Theorem 4 Suppose there always exists a mapping ξ to map h ( f ( X )) to g ( f ( X )) . 
Revised: Theorem 4 Suppose there always exists a mapping ξ to map h ( f ( X )) to g ( f ( X )) . I [Figure]

Original: I.e., g ( f ( X )) =
Revised: e., g ( f ( X )) =

Original: ξ ( h ( f ( X ))) .
Revised: ξ ( h ( f ( X ))) .

Original: Then minimizing Equation (5) is implicitly maximizing Mutual Information between h ( f ( X )) and g ( f ( X )) .
Revised: Then minimizing Equation (5) is implicitly maximizing Mutual Information between h ( f ( X )) and g ( f ( X )) .

Original: I.e., when Equation (5) achieves 0 , the mutual information
Revised: I.e., when Equation (5) achieves 0 , the mutual information

Original: I ( h ( f ( X )) , g ( f ( X ))) achieves maximum.
Revised: I ( h ( f ( X )) , g ( f ( X ))) achieves maximum.

Original: The above results facilitate a better understanding on what the regularizer is exactly doing.
Revised: The above results facilitate a better understanding on what the regularizer is exactly doing.

Original: Note that
Revised: Note that

Original: Mutual Information itself has several popular estimators (Belghazi et al., 2018; Hjelm et al., 2018).
Revised: Mutual Information itself has several popular estimators (Belghazi et al., 2018; Hjelm et al., 2018).

Original: It is a very interesting future direction to develop regularizes based on MI to perform regularization by utilizing SSL features.
Revised: It is a very interesting future direction to develop regularizes based on MI to perform regularization by utilizing SSL features.

Original: 4. E XPERIMENTS
Revised: 4. E XPERIMENTS

Original: We provide experiments on CIFAR10 to verify our theory and analysis.
Revised: We provide experiments on CIFAR10 to verify our theory and analysis.

Original: The overall experiments are shown in Figure 5.
Revised: The overall experiments are shown in Figure 5.

Original: In the experiments, Regularizer is added at the very beginning since recent studies show that for random initialized network, the model tends to ﬁt clean labels ﬁrst (Arpit et al., 2017) and we hope the regularizer can improve the network robustness when DNN begins to ﬁt noisy labels.
Revised: In the experiments, Regularizer is added at the very beginning since recent studies show that for random initialized network, the model tends to ﬁt clean labels ﬁrst (Arpit et al., 2017) and we hope the regularizer can improve the network robustness when DNN begins to ﬁt noisy labels.

Original: From Figure 5 (c) (d), for CE training, the performance ﬁrst increases then decreases since the network over-ﬁts noisy labels as training proceeds.
Revised: From Figure 5 (c) (d), for CE training, the performance ﬁrst increases then decreases since the network over-ﬁts noisy labels as training proceeds.

Original: However, for CE with regularizer, the performance is more stable after it reaches the peak.
Revised: However, for CE with regularizer, the performance is more stable after it reaches the peak.

Original: For 60% noise rate, the peak point is also much higher than vanilla CE training.
Revised: For 60% noise rate, the peak point is also much higher than vanilla CE training.

Original: For Figure 5 (a) (b), since the network is not randomly initialized, it over-ﬁts noisy labels at the very beginning and the performance gradually decreases.
Revised: For Figure 5 (a) (b), since the network is not randomly initialized, it over-ﬁts noisy labels at the very beginning and the performance gradually decreases.

Original: However, for CE with regularizer, it can help the network gradually increase the performance as the network reaches the lowest point (over-ﬁtting state).
Revised: However, for CE with regularizer, it can help the network gradually increase the performance as the network reaches the lowest point (over-ﬁtting state).

Original: This observation supports Theorem 3 that the regularizer can prevent DNN from over-ﬁtting to noisy labels.
Revised: This observation supports Theorem 3 that the regularizer can prevent DNN from over-ﬁtting to noisy labels.

Original: More experiments and ablation studies can be found in the Appendix.
Revised: More experiments and ablation studies can be found in the Appendix.

Original:  C ONCLUSIONS AND D ISCUSSIONS
Revised:  C ONCLUSIONS AND D ISCUSSIONS

Original: In this paper, we have provided a theoretical understanding on why and how self-supervised features improve training from noisy labels and perform experiments to verify our analysis.
Revised: In this paper, we have provided a theoretical understanding on why and how self-supervised features improve training from noisy labels and perform experiments to verify our analysis.

Original: • We show that with SSL pre-trained encoder, CE itself is theoretically robust to symmetric label noise.
Revised: • We show that with SSL pre-trained encoder, CE itself is theoretically robust to symmetric label noise.

Original: With the down-sampling strategy, it can also exhibit great power on class- and instancedependent label noise.
Revised: With the down-sampling strategy, it can also exhibit great power on class- and instancedependent label noise.

Original: This part answers why SSL features help (Theorem 1–2).
Revised: This part answers why SSL features help (Theorem 1–2).

Original: • We show that by utilizing SSL features, the regularizer in KD can help DNN alleviate the overﬁtting problem on the noisy dataset and build theoretical insights on how SSL features relate to the network robustness.
Revised: • We show that by utilizing SSL features, the regularizer in KD can help DNN alleviate the overﬁtting problem on the noisy dataset and build theoretical insights on how SSL features relate to the network robustness.

Original: This part answers how SSL features help (Theorem 3–4).
Revised: This part answers how SSL features help (Theorem 3–4).

Original: Future directions can be done to develop more efﬁcient sampling strategies for solving instancebased label noise with theoretical guarantees and develop efﬁcient regularizers to combat label noise.
Revised: Future directions can be done to develop more efﬁcient sampling strategies for solving instancebased label noise with theoretical guarantees and develop efﬁcient regularizers to combat label noise.

Original: The key for these future works is to leverage the nice structural property of SSL features.
Revised: The key for these future works is to leverage the nice structural property of SSL features.

Original: A PPENDIX
Revised: A PPENDIX

Original: The Appendix is arranged as follows:
Revised: The Appendix is arranged as follows:

Original: Section A proves Theorem 1 and Theorem 2 in the main paper.
Revised: Section A proves Theorem 1 and Theorem 2 in the main paper.

Original: Section B proves Theorem 3 and Theorem 4 in the main Paper.
Revised: Section B proves Theorem 3 and Theorem 4 in the main Paper.

Original: Section C illustrates why down-sampling can decrease the gap of noise rates.
Revised: Section C illustrates why down-sampling can decrease the gap of noise rates.

Original: Section D provides additional discussions and experiments including further discussion of whether or not to ﬁx the encoder when ﬁne-tuning DNN on noisy dataset; analyses of why robust losses (Wang et al., 2019; Ma et al., 2020) may suffer severe performance degradation under high noise rates; the effect of distance measure in Equation (4) ( w = 1 or 2); ablation study in Section 4; visualization of features trained by vanilla CE loss.
Revised: Section D provides additional discussions and experiments including further discussion of whether or not to ﬁx the encoder when ﬁne-tuning DNN on noisy dataset; analyses of why robust losses (Wang et al., 2019; Ma et al., 2020) may suffer severe performance degradation under high noise rates; the effect of distance measure in Equation (4) ( w = 1 or 2); ablation study in Section 4; visualization of features trained by vanilla CE loss; the effect of different SSL pre-trained methods; Experiments with human-annotated label noise.

Original: Section E elaborates the detailed experimental setting of all the experiments in the paper.
Revised: Section E elaborates the detailed experimental setting of all the experiments in the paper.

Revised: Section F generalizes Theorem 1 to multi-class classiﬁcation.

Revised: Section G provides theoretical and experimental proof on why directly ﬁne-tuning SSL features cannot be robust on asymmetric and instance label noise.

Original: A P ROOF FOR T HEOREM 1, A. P ROOF FOR T HEOREM
Revised: A P ROOF FOR T HEOREM 1, A. P ROOF FOR T HEOREM

Original: Deﬁne g 1 = min
Revised: Deﬁne g 1 = min

Original: g E D [ CE ( g ( f ( X )) , Y )] and g 2 =
Revised: g E D [ CE ( g ( f ( X )) , Y )] and g 2 =

Original: min g E (cid:101) D [ CE ( g ( f ( X ))) , (cid:101) Y )] .
Revised: min g E (cid:101) D [ CE ( g ( f ( X ))) , (cid:101) Y )] .

Original: Suppose g Bayesand g Bayesare Bayes optimal classiﬁers under P ( f ( X ) , Y ) and P ( f ( X ) , (cid:101) Y ) , respectively.
Revised: Suppose g Bayesand g Bayesare Bayes optimal classiﬁers under P ( f ( X ) , Y ) and P ( f ( X ) , (cid:101) Y ) , respectively.

Original: Since CE is a calibrated and convex loss, with enough samples, g 1 and g 2 will converge to (approximate to)
Revised: Since CE is a calibrated and convex loss, with enough samples, g 1 and g 2 will converge to (approximate to)

Original: g Bayesand g Bayes(Bartlett et
Revised: g Bayesand g Bayes(Bartlett et

Original: al., 2006).
Revised: al., 2006).

Original: Thus to prove g 1 and g 2 have the same prediction for f ( X ) , it is identical to prove g Bayesand g Bayeshave the same predictions for f ( X ) .
Revised: Thus to prove g 1 and g 2 have the same prediction for f ( X ) , it is identical to prove g Bayesand g Bayeshave the same predictions for f ( X ) .

Original: Let p + ( x ) be the distribution of f ( X + ) and p − ( x ) be the distribution of f ( X − ) , according to the bayes rules:[Equation]
Revised: Let p + ( x ) be the distribution of f ( X + ) and p − ( x ) be the distribution of f ( X − ) , according to the bayes rules:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ψ ( x ) =
Revised: Deﬁne ψ ( x ) =

Original: P ( Y = 1) · p + ( x ) − P
Revised: P ( Y = 1) · p + ( x ) − P

Original: ( Y = 0) · p − ( x ) .
Revised: ( Y = 0) · p − ( x ) .

Original: Thus ψ ( x ) = 0 is the decision boundary for optimal Bayes classiﬁer g Bayeson clean dataset.
Revised: Thus ψ ( x ) = 0 is the decision boundary for optimal Bayes classiﬁer g Bayeson clean dataset.

Original: Since when ψ ( x ) > 0 , P ( Y = 1 | x ) > P ( Y = 

Revised: Since when ψ ( x ) > 0 , P ( Y = 1 | x ) > P ( Y =

Original: | x ) from Equation (8) (9), the classiﬁer will output the prediction as 1 and vice versa.
Revised: | x ) from Equation (8) (9), the classiﬁer will output the prediction as 1 and vice versa.

Original: Since the generation of label noise is independent to f ( X ) , the samples for each class whose labels are ﬂipped also follows original distribution.
Revised: Since the generation of label noise is independent to f ( X ) , the samples for each class whose labels are ﬂipped also follows original distribution.

Original: Denote p e + ( x ) to be the distribution of samples whose observed labels are 1 under label noise.
Revised: Denote p e + ( x ) to be the distribution of samples whose observed labels are 1 under label noise.

Original: Thus p e + ( x ) and p e − ( x ) can be represented as:
Revised: Thus p e + ( x ) and p e − ( x ) can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where m 1 and m 2 are normalization term to make distribution valid.
Revised: where m 1 and m 2 are normalization term to make distribution valid.

Original: Speciﬁcally, m 1 =
Revised: Speciﬁcally, m 1 =

Original: P ( Y =1) · (1 − e + )+ P ( Y =0) · e − and m 2 = 1 P ( Y =1) · e + + P ( Y =0) · (1 − e − ) .
Revised: P ( Y =1) · (1 − e + )+ P ( Y =0) · e − and m 2 = 1 P ( Y =1) · e + + P ( Y =0) · (1 − e − ) .

Original: Under label noise, P ( (cid:101) Y = 1) =
Revised: Under label noise, P ( (cid:101) Y = 1) =

Original: P ( Y = 1) · (1 − e + ) + P ( Y = 0) · e − and P ( (cid:101) Y = 0)
Revised: P ( Y = 1) · (1 − e + ) + P ( Y = 0) · e − and P ( (cid:101) Y = 0)

Original: =
Revised: =

Original: P ( Y = 1) · e + + P ( Y = 0) · (1 − e − ) .
Revised: P ( Y = 1) · e + + P ( Y = 0) · (1 − e − ) .

Original: According bayes rules, we have:
Revised: According bayes rules, we have:

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we assume there exits a x 0 in the feature space which satisfy ψ ( x 0 ) > 0 .
Revised: Without loss of generality, we assume there exits a x 0 in the feature space which satisfy ψ ( x 0 ) > 0 .

Original: Thus for clean dataset, P ( Y = 1) · p + ( x ) > P ( Y = 0) · p − ( x ) and P ( Y = 1 | x 0 ) >
Revised: Thus for clean dataset, P ( Y = 1) · p + ( x ) > P ( Y = 0) · p − ( x ) and P ( Y = 1 | x 0 ) >

Original: P ( Y = 

Revised: P ( Y =

Original: | x 0 ) .
Revised: | x 0 ) .

Original: We aim to show that for noisy dataset, P ( (cid:101) Y = 1 | x 0 ) >
Revised: We aim to show that for noisy dataset, P ( (cid:101) Y = 1 | x 0 ) >

Original: P ( (cid:101) Y = 

Revised: P ( (cid:101) Y =

Original: | x 0 ) , which means ψ ( x ) = 

Revised: | x 0 ) , which means ψ ( x ) =

Original: is also the decision boundary that gives Bayes optimal classiﬁer g Bayesunder label noise.
Revised: is also the decision boundary that gives Bayes optimal classiﬁer g Bayesunder label noise.

Original: According to Equation (10) and Equation (11),
Revised: According to Equation (10) and Equation (11),

Original: [Equation]
Revised: [Equation]

Original: normalization
Revised: normalization

Original: [Equation]
Revised: [Equation]

Original: 12) where normalization =
Revised: where normalization =

Original: P ( (cid:101) Y = 1) · p e + ( x ) + P ( (cid:101) Y = 0) · p e − ( x ) > 0 .
Revised: P ( (cid:101) Y = 1) · p e + ( x ) + P ( (cid:101) Y = 0) · p e − ( x ) > 0 .

Original: Since P ( Y = 1) · p + ( x 0 ) >
Revised: Since P ( Y = 1) · p + ( x 0 ) >

Original: P ( Y = 0) · p − ( x 0 ) , if e + = e − < 0 .
Revised: P ( Y = 0) · p − ( x 0 ) , if e + = e − < 0 .

Original: 5 , we have P ( (cid:101) Y = 1 | x 0 ) − P ( (cid:101)
Revised: 5 , we have P ( (cid:101) Y = 1 | x 0 ) − P ( (cid:101)

Original: Y = 

Revised: Y =

Original: | x 0 ) > 0 .
Revised: | x 0 ) > 0 .

Original: Thus ψ ( x ) =is also the decision boundary of g Bayes.
Revised: Thus ψ ( x ) =is also the decision boundary of g Bayes.

Original: g Bayesand g
Revised: g Bayesand g

Original: Bayeswill output the same prediction for all the samples.
Revised: Bayeswill output the same prediction for all the samples.

Original: Proof done.
Revised: Proof done.

Original: A. P ROOF FOR T HEOREM
Revised: A. P ROOF FOR T HEOREM

Original: Let d = || µ 1 − µ 2 || .
Revised: Let d = || µ 1 − µ 2 || .

Original: Without loss of generality, we assume µ 1 = ( − d 2 , 0 , . . . , 0) and µ 2 = ( d 2 , 0 , . . . , 0)
Revised: Without loss of generality, we assume µ 1 = ( − d 2 , 0 , . . . , 0) and µ 2 = ( d 2 , 0 , . . . , 0)

Original: (This can be done by applying afﬁne transformation on original coordinates without changing actual positions of f ( X + ) and f ( X − ) ).
Revised: (This can be done by applying afﬁne transformation on original coordinates without changing actual positions of f ( X + ) and f ( X − ) ).

Original: Under the distribution of Gaussian assumption, we write the distribution of f ( X + ) as follows:
Revised: Under the distribution of Gaussian assumption, we write the distribution of f ( X + ) as follows:

Original: [Equation]
Revised: [Equation]

Original: (13) where x = ( x 1 , . . 
Revised: 13) where x = ( x 1 , . . .

Original: , x k ) T ∈ R k , x k denotes the variable of x in the k -th dimension.
Revised: , x k ) T ∈ R k , x k denotes the variable of x in the k -th dimension.

Original: Similarly, the pdf of f ( X − ) can be represented as:
Revised: Similarly, the pdf of f ( X − ) can be represented as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Following notations from A.1, solving ψ ( x ) = 

Revised: Following notations from A.1, solving ψ ( x ) =

Original: under the conditions of P ( Y = 1)
Revised: under the conditions of P ( Y = 1)

Original: =
Revised: =

Original: P ( Y = 0) gives x 1 = 0 ( x 1 is the variable of x in the ﬁrst dimension).
Revised: P ( Y = 0) gives x 1 = 0 ( x 1 is the variable of x in the ﬁrst dimension).

Original: Thus, when x 1 < 0 , the Bayes optimal classiﬁer f ◦ g Bayes outputs prediction as 1 . I.e., for samples whose labels are 1 , f ◦ g Bayes gives right prediction when x 1 < 0 and wrong prediction when x 1 > 0 .
Revised: Thus, when x 1 < 0 , the Bayes optimal classiﬁer f ◦ g Bayes outputs prediction as 1 . I.e., for samples whose labels are 1 , f ◦ g Bayes gives right prediction when x 1 < 0 and wrong prediction when x 1 > 0 .

Original: The risk of of f ◦ g Bayes can be written as:
Revised: The risk of of f ◦ g Bayes can be written as:

Original: [Equation]
Revised: [Equation]

Original: (a) is satisﬁed because of the symmetry of p + ( x ) and p − ( x ) .
Revised: (a) is satisﬁed because of the symmetry of p + ( x ) and p − ( x ) .

Original: We derive (b) as follows:
Revised: We derive (b) as follows:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:82) x1 √
Revised: Since (cid:82) x1 √

Original: [Equation]
Revised: [Equation]

Original: exp {
Revised: exp {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 2 σ} d x 1 is one dimensional gaussian with mean − d 2 and deviation σ , it is easy to calculate that (cid:82) x 1 <
Revised: 2 2 σ} d x 1 is one dimensional gaussian with mean − d 2 and deviation σ , it is easy to calculate that (cid:82) x 1 <

Original: [Equation]
Revised: [Equation]

Original: Φ
Revised: Φ

Original: [Equation]
Revised: [Equation]

Original: σ ) , where Φ is the cdf of standard gaussian distribution.
Revised: σ ) , where Φ is the cdf of standard gaussian distribution.

Original: Thus (cid:82) x
Revised: Thus (cid:82) x

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: Equation (b) is satisﬁed.
Revised: Equation (b) is satisﬁed.

Original: Proof done.
Revised: Proof done.

Original: B P ROOF FOR T HEOREM 3, 
Revised: B P ROOF FOR T HEOREM 3,

Original: Lemma 1 If X and Y are independent and follow gaussian distribution: X ∼ N ( µ X , Σ X ) and
Revised: Lemma 1 If X and Y are independent and follow gaussian distribution: X ∼ N ( µ X , Σ X ) and

Original: Y ∼ N ( µ Y , Σ Y ) , Then: E X,Y ( || X − Y || 2 ) =
Revised: Y ∼ N ( µ Y , Σ Y ) , Then: E X,Y ( || X − Y || 2 ) =

Original: || µ X − µ Y || 2 + tr (Σ X + Σ Y ) .
Revised: || µ X − µ Y || 2 + tr (Σ X + Σ Y ) .

Original: B. P ROOF FOR T HEOREM
Revised: B. P ROOF FOR T HEOREM

Original: Before the derivation, we deﬁne some notations for better presentation.
Revised: Before the derivation, we deﬁne some notations for better presentation.

Original: Deﬁne the labels of X clean as Y clean and the labels of X noisy as Y noisy .
Revised: Deﬁne the labels of X clean as Y clean and the labels of X noisy as Y noisy .

Original: Under the label noise, it is easy to verify P ( Y clean = 1) =
Revised: Under the label noise, it is easy to verify P ( Y clean = 1) =

Original: P ( Y =1) · (1 − e + )
Revised: P ( Y =1) · (1 − e + )

Original: P ( Y =1) · (1 − e + )+ P ( Y =0) · (1 − e − ) and P ( Y noisy = 1) =
Revised: P ( Y =1) · (1 − e + )+ P ( Y =0) · (1 − e − ) and P ( Y noisy = 1) =

Original: P ( Y =0) · e − P
Revised: P ( Y =0) · e − P

Original: ( Y =0) ·
Revised: ( Y =0) ·

Original: e − + P ( Y =1) · e + .
Revised: e − + P ( Y =1) · e + .

Original: Let p 1 =
Revised: Let p 1 =

Original: P
Revised: P

Original: ( Y clean = 1) , p 2 =
Revised: ( Y clean = 1) , p 2 =

Original: P ( Y noisy = 1) , g ( f ( X )) and h ( f ( X )) to be simpliﬁed as gf ( X ) and hf ( X ) .
Revised: P ( Y noisy = 1) , g ( f ( X )) and h ( f ( X )) to be simpliﬁed as gf ( X ) and hf ( X ) .

Original: In the case of binary classiﬁcation, gf ( x ) is one dimensional value which denotes the network prediction on x belonging to Y = 1 .
Revised: In the case of binary classiﬁcation, gf ( x ) is one dimensional value which denotes the network prediction on x belonging to Y = 1 .

Original: L c can be written as:
Revised: L c can be written as:

Original: [Equation]
Revised: [Equation]

Original: ) where m 1 and m 2 are normalization terms from Equation (4).
Revised: ) where m 1 and m 2 are normalization terms from Equation (4).

Original: (a) is satisﬁed because Ψ ( X clean , X noisy ) is irrelevant to the labels.
Revised: (a) is satisﬁed because Ψ ( X clean , X noisy ) is irrelevant to the labels.

Original: We derive Ψ ( X clean + , X noisy + ) as follows:
Revised: We derive Ψ ( X clean + , X noisy + ) as follows:

Original: [Equation]
Revised: [Equation]

Original: (b) is satisﬁed because from Assumption 2, DNN has conﬁdent prediction on clean samples.
Revised: (b) is satisﬁed because from Assumption 2, DNN has conﬁdent prediction on clean samples.

Original: (c) is satisﬁed because g ( f ( x )) is one dimensional value which ranges from 0 to 1.
Revised: (c) is satisﬁed because g ( f ( x )) is one dimensional value which ranges from 0 to 1.

Original: From Assumption 4, hf ( X + ) and hf ( X − ) follows gaussian distribution with parameter ( µ 1 , Σ) and ( µ 2 , Σ) .
Revised: From Assumption 4, hf ( X + ) and hf ( X − ) follows gaussian distribution with parameter ( µ 1 , Σ) and ( µ 2 , Σ) .

Original: Thus according to Lemma 1, we have E X clean + ,X noisy
Revised: Thus according to Lemma 1, we have E X clean + ,X noisy

Original: + ||
Revised: + ||

Original: hf
Revised: hf

Original: ( X clean + ) − hf
Revised: ( X clean + ) − hf

Original: ( X noisy + )
Revised: ( X noisy + )

Original: [Equation]
Revised: [Equation]

Original: Similarly, one can calculate E X clean − ,X noisy
Revised: Similarly, one can calculate E X clean − ,X noisy

Original: + ||
Revised: + ||

Original: hf
Revised: hf

Original: ( X clean − ) − hf
Revised: ( X clean − ) − hf

Original: ( X noisy + )
Revised: ( X noisy + )

Original: || 2 = 2 · tr (Σ) .
Revised: || 2 = 2 · tr (Σ) .

Original: It can be seen that (d) is function with respect to gf ( X noisy + ) .
Revised: It can be seen that (d) is function with respect to gf ( X noisy + ) .

Original: Similarly, Ψ ( X clean − , X noisy + ) is also a function with respect to gf ( X noisy + ) while Ψ ( X clean + , X noisy − ) and Ψ ( X clean − , X noisy − ) are functions with respect to gf ( X noisy − ) .
Revised: Similarly, Ψ ( X clean − , X noisy + ) is also a function with respect to gf ( X noisy + ) while Ψ ( X clean + , X noisy − ) and Ψ ( X clean − , X noisy − ) are functions with respect to gf ( X noisy − ) .

Original: Denote d (+ , +) =
Revised: Denote d (+ , +) =

Original: E X clean + ,X noisy
Revised: E X clean + ,X noisy

Original: + ||
Revised: + ||

Original: hf
Revised: hf

Original: ( X clean + ) − hf
Revised: ( X clean + ) − hf

Original: ( X noisy + )
Revised: ( X noisy + )

Original: || 2 .
Revised: || 2 .

Original: After organizing Ψ ( X clean + , X noisy + ) and Ψ ( X clean − , X noisy + ) , we have:
Revised: After organizing Ψ ( X clean + , X noisy + ) and Ψ ( X clean − , X noisy + ) , we have:

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: constant with respect to gf ( X
Revised: constant with respect to gf ( X

Original: [Equation]
Revised: [Equation]

Original: + )
Revised: + )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note in Equation (15), we use ( E X noisy
Revised: Note in Equation (15), we use ( E X noisy

Original: + gf ( X noisy + ))
Revised: + gf ( X noisy + ))

Original: 2 to approximate E X noisy
Revised: 2 to approximate E X noisy

Original: + gf ( X noisy + ) 2 since from
Revised: + gf ( X noisy + ) 2 since from

Original: Assumption 3, var ( g ( f ( X noisy + )))
Revised: Assumption 3, var ( g ( f ( X noisy + )))

Original: → 0 .
Revised: → 0 .

Original: Now we calculate m 1 and m 2 from Equation (4):
Revised: Now we calculate m 1 and m 2 from Equation (4):

Original: [Equation]
Revised: [Equation]

Original: Under the condition of P ( Y = 1)
Revised: Under the condition of P ( Y = 1)

Original: =
Revised: =

Original: P ( Y = 0) , e − =
Revised: P ( Y = 0) , e − =

Original: e + , we have p 1 = p 2 = 12 , m
Revised: e + , we have p 1 = p 2 = 12 , m

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: tr
Revised: tr

Original: [Equation]
Revised: [Equation]

Original: + ||
Revised: + ||

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: , m 1 = 12 , which is constant with respect to E X noisy
Revised: , m 1 = 12 , which is constant with respect to E X noisy

Original: + gf ( X noisy
Revised: + gf ( X noisy

Original: + ) and
Revised: + ) and

Original: E X noisy − gf
Revised: E X noisy − gf

Original: ( X noisy − ) in Equation (16).
Revised: ( X noisy − ) in Equation (16).

Original: Thus Equation (15) is a quadratic equation with respect to
Revised: Thus Equation (15) is a quadratic equation with respect to

Original: E X noisy
Revised: E X noisy

Original: + gf ( X noisy + ) .
Revised: + gf ( X noisy + ) .

Original: Then when Equation (15) achieves global minimum, we have:
Revised: Then when Equation (15) achieves global minimum, we have:

Original: [Equation]
Revised: [Equation]

Original: Similarly, organizing Ψ ( X clean + , X noisy − ) and Ψ ( X clean − , X noisy − ) gives the solution of E X noisy − gf
Revised: Similarly, organizing Ψ ( X clean + , X noisy − ) and Ψ ( X clean − , X noisy − ) gives the solution of E X noisy − gf

Original: ( X noisy − ) :
Revised: ( X noisy − ) :

Original: [Equation]
Revised: [Equation]

Original: Proof Done.
Revised: Proof Done.

Original: B. P ROOF FOR T HEOREM
Revised: B. P ROOF FOR T HEOREM

Original: We ﬁrst refer to a property of Mutual Information:
Revised: We ﬁrst refer to a property of Mutual Information:

Original: [Equation]
Revised: [Equation]

Original: 19) where ψ and φ are any invertible functions.
Revised: ) where ψ and φ are any invertible functions.

Original: This perporty shows that mutual information is invariant to invertible transformations (Cover, 1999).
Revised: This property shows that mutual information is invariant to invertible transformations (Cover, 1999).

Original: Thus to prove the theorem, we only need to prove that ξ in Theorem 4 must be an invertible function when Equation (5) is minimized to 0.
Revised: Thus to prove the theorem, we only need to prove that ξ in Theorem 4 must be an invertible function when Equation (5) is minimized to 0.

Original: Since when ξ is invertible, I ( h ( f ( X )) , g ( f ( X )))
Revised: Since when ξ is invertible, I ( h ( f ( X )) , g ( f ( X )))

Original: =
Revised: =

Original: I ( h ( f ( X )) , ξ ( h ( f ( X ))))
Revised: I ( h ( f ( X )) , ξ ( h ( f ( X ))))

Original: =
Revised: =

Original: I ( h ( f ( X )) , h ( f ( X ))) .
Revised: I ( h ( f ( X )) , h ( f ( X ))) .

Original: We prove this by contradiction.
Revised: We prove this by contradiction.

Original: Let t i
Revised: Let t i

Original: = h
Revised: = h

Original: ( f ( x i ))
Revised: ( f ( x i ))

Original: and s i =
Revised: and s i =

Original: g ( f ( x i )) .
Revised: g ( f ( x i )) .

Original: Suppose ξ is not invertible, then there must exists s i and s j where s i (cid:54)
Revised: Suppose ξ is not invertible, then there must exists s i and s j where s i (cid:54)

Original: =
Revised: =

Original: s j which satisfy t j = ξ ( s i )
Revised: s j which satisfy t j = ξ ( s i )

Original: =
Revised: =

Original: t i .
Revised: t i .

Original: However, under this condition, t i − t j = 

Revised: However, under this condition, t i − t j =

Original: and s i − s j (cid:54) = 0 , Equation (5) can not be minimized to 0.
Revised: and s i − s j (cid:54) = 0 , Equation (5) can not be minimized to 0.

Original: Thus when Equation (5) is minimized to 0, ξ must be an invertible function.
Revised: Thus when Equation (5) is minimized to 0, ξ must be an invertible function.

Original: Proof done.
Revised: Proof done.

Original: B. P ROOF FOR L EMMA
Revised: B. P ROOF FOR L EMMA

Original: By the independence condition, Z = X − Y also follows gaussian distribution with parameter ( µ X − µ Y , Σ X + Σ Y ) .
Revised: By the independence condition, Z = X − Y also follows gaussian distribution with parameter ( µ X − µ Y , Σ X + Σ Y ) .

Original: Write Z as Z = µ + LU where U is a standard gaussian and µ = µ X − µ Y , LL T = Σ X + Σ Y .
Revised: Write Z as Z = µ + LU where U is a standard gaussian and µ = µ X − µ Y , LL T = Σ X + Σ Y .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Since U is standard gaussian, E ( U ) = 0 .
Revised: Since U is standard gaussian, E ( U ) = 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: (a) is satisﬁed because U is standard gaussian, thus E ( U 2 k )
Revised: (a) is satisﬁed because U is standard gaussian, thus E ( U 2 k )

Original: = 1 and E
Revised: = 1 and E

Original: ( U k U l ) = 

Revised: ( U k U l ) =

Original: ( k (cid:54) = l ) .
Revised: ( k (cid:54) = l ) .

Original: Proof Done.
Revised: Proof Done.

Original: C I LLUSTRATING DOWN - SAMPLING STRATEGY
Revised: C I LLUSTRATING DOWN - SAMPLING STRATEGY

Original: We illustrate in the case of binary classiﬁcation with e + + e − < 1 .
Revised: We illustrate in the case of binary classiﬁcation with e + + e − < 1 .

Original: Suppose the dataset is balanced, at the initial sate, e + > e − .
Revised: Suppose the dataset is balanced, at the initial sate, e + > e − .

Original: After down-sampling, the noise rate becomes e ∗ + and e ∗− .
Revised: After down-sampling, the noise rate becomes e ∗ + and e ∗− .

Original: We aim to prove two propositions:
Revised: We aim to prove two propositions:

Original: Proposition 1 If e + and e − are known, the optimal down-sampling rate can be calculated by e + and e − to make e ∗ + = e ∗−
Revised: Proposition 1 If e + and e − are known, the optimal down-sampling rate can be calculated by e + and e − to make e ∗ + = e ∗−

Original: Proposition 2 If e + and e − are not known.
Revised: Proposition 2 If e + and e − are not known.

Original: When down-sampling strategy is to make P ( (cid:101) Y = 1) =
Revised: When down-sampling strategy is to make P ( (cid:101) Y = 1) =

Original: P ( (cid:101) Y = 0) , then 0 < e ∗ + − e ∗− <
Revised: P ( (cid:101) Y = 0) , then 0 < e ∗ + − e ∗− <

Original: e + − e − .
Revised: e + − e − .

Original: Proof for Proposition 1: Since dataset is balanced with initial e + > e − , we have P ( (cid:101) Y = 1) <
Revised: Proof for Proposition 1: Since dataset is balanced with initial e + > e − , we have P ( (cid:101) Y = 1) <

Original: P ( (cid:101) Y = 0) .
Revised: P ( (cid:101) Y = 0) .

Original: Thus down-sampling is conducted at samples whose observed label are 0 .
Revised: Thus down-sampling is conducted at samples whose observed label are 0 .

Original: Suppose the random down-sampling rate is r , then e ∗ + = r · e + 1 − e + + r · e + and e ∗− =
Revised: Suppose the random down-sampling rate is r , then e ∗ + = r · e + 1 − e + + r · e + and e ∗− =

Original: e − r ·
Revised: e − r ·

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: + e − .
Revised: + e − .

Original: If e ∗ + = e ∗− , we have:
Revised: If e ∗ + = e ∗− , we have:

Original: [Equation]
Revised: [Equation]

Original: Thus the optimal down-sampling rate r = (cid:113) e − · (1 − e + ) e + · (1 − e − ) , which can be calculated if e − and e + are known.
Revised: Thus the optimal down-sampling rate r = (cid:113) e − · (1 − e + ) e + · (1 − e − ) , which can be calculated if e − and e + are known.

Original: Proof for Proposition 2: If down sampling strategy is to make P ( (cid:101) Y = 1) =
Revised: Proof for Proposition 2: If down sampling strategy is to make P ( (cid:101) Y = 1) =

Original: P ( (cid:101) Y = 0) , then r · ( e + + 1 − e − ) =
Revised: P ( (cid:101) Y = 0) , then r · ( e + + 1 − e − ) =

Original: 1 − e + + e − , we have r = 1 − e + +
Revised: 1 − e + + e − , we have r = 1 − e + +

Original: e
Revised: e

Original: − 1 − e − + e + .
Revised: − 1 − e − + e + .

Original: Thus e ∗ + can be calculated as:
Revised: Thus e ∗ + can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: Denote α =
Revised: Denote α =

Original: [Equation]
Revised: [Equation]

Original: Since e + > e − , 1 − e −
Revised: Since e + > e − , 1 − e −

Original: + e +
Revised: + e +

Original: [Equation]
Revised: [Equation]

Original: 1 − e + + e − , α =
Revised: 1 − e + + e − , α =

Original: [Equation]
Revised: [Equation]

Original: Similarly, e ∗− can be calculated as:
Revised: Similarly, e ∗− can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: Denote β = 1 − e − + e + e − · (1 − e − + e + )+(1 − e − ) · (1 − e + + e − ) .
Revised: Denote β = 1 − e − + e + e − · (1 − e − + e + )+(1 − e − ) · (1 − e + + e − ) .

Original: Since e + > e − , 1 − e −
Revised: Since e + > e − , 1 − e −

Original: + e + > 1 − e + + e − , β = 1 − e − +
Revised: + e + > 1 − e + + e − , β = 1 − e − +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: > 1 − e − + e + e − · (1 − e − + e + )+(1 − e − ) · (1 − e − + e + )
Revised: > 1 − e − + e + e − · (1 − e − + e + )+(1 − e − ) · (1 − e − + e + )

Original: = 1 .
Revised: = 1 .

Original: Since α · e + < e + and β · e − > e − , we have e ∗ +
Revised: Since α · e + < e + and β · e − > e − , we have e ∗ +

Original: −
Revised: −

Original: e ∗− =
Revised: e ∗− =

Original: α · e + − β · e − < e + − e − .
Revised: α · e + − β · e − < e + − e − .

Original: Next, we prove e ∗ + > e ∗− , following the derivation below:
Revised: Next, we prove e ∗ + > e ∗− , following the derivation below:

Original: [Equation]
Revised: [Equation]

Original: Let f ( e + ) =
Revised: Let f ( e + ) =

Original: e + · (1 − e + ) + e + · e 2 − 1 − e + − e − · (1 − e − ) − e − · e 2+ 1 − e − .
Revised: e + · (1 − e + ) + e + · e 2 − 1 − e + − e − · (1 − e − ) − e − · e 2+ 1 − e − .

Original: Since we have assumed e − < e + and e − + e + < 1 .
Revised: Since we have assumed e − < e + and e − + e + < 1 .

Original: Thus proving e ∗ + > e ∗− is identical to prove f ( e + ) > 0 when e − < e + < 1 − e − .
Revised: Thus proving e ∗ + > e ∗− is identical to prove f ( e + ) > 0 when e − < e + < 1 − e − .

Original: Firstly, it is easy to verify when e + = e − or e + = 1 − e − , f ( e + ) = 0 .
Revised: Firstly, it is easy to verify when e + = e − or e + = 1 − e − , f ( e + ) = 0 .

Original: From Mean Value Theory, there must exists a point e 0 which satisfy f (cid:48)
Revised: From Mean Value Theory, there must exists a point e 0 which satisfy f (cid:48)

Original: ( e 0 )
Revised: ( e 0 )

Original: = 

Revised: =

Original: where e + < e 0 < 1 − e − .
Revised: where e + < e 0 < 1 − e − .

Original: Next, we differentiate
Revised: Next, we differentiate

Original: [Figure]
Revised: [Figure]

Original: e_ = 0. e_ = 0.
Revised: e_ = 0. e_ = 0.

Original: f ( e + ) as follows:
Revised: f ( e + ) as follows:

Original: [Equation]
Revised: [Equation]

Original: It can be veriﬁed that f (cid:48) ( e − ) =
Revised: It can be veriﬁed that f (cid:48) ( e − ) =

Original: [Equation]
Revised: [Equation]

Original: > 0 and f (cid:48) (1 − e − ) =
Revised: > 0 and f (cid:48) (1 − e − ) =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Further differentiate f (cid:48)
Revised: Further differentiate f (cid:48)

Original: ( e + ) , we get when e + < 1 − ((1 − e − ) · e 2 − ) 13 , f (cid:48)(cid:48)
Revised: ( e + ) , we get when e + < 1 − ((1 − e − ) · e 2 − ) 13 , f (cid:48)(cid:48)

Original: ( e + ) < 0 and when e + > 1 − ((1 − e − ) · e 2 − ) 13 , f (cid:48)(cid:48)
Revised: ( e + ) < 0 and when e + > 1 − ((1 − e − ) · e 2 − ) 13 , f (cid:48)(cid:48)

Original: ( e + ) > 0 .
Revised: ( e + ) > 0 .

Original: Since e − < e + and e − + e + < 1 , we have e − < 12 and e − < 1 − ((1 − e − ) · e 2 − ) 13 < 1 − e − , i.e., 1 − ((1 − e − ) · e 2 − ) 13 locates in the point between e − and 1 − e − .
Revised: Since e − < e + and e − + e + < 1 , we have e − < 12 and e − < 1 − ((1 − e − ) · e 2 − ) 13 < 1 − e − , i.e., 1 − ((1 − e − ) · e 2 − ) 13 locates in the point between e − and 1 − e − .

Original: Thus, when e − < e + < 1 − ((1 − e − ) · e 2 − ) 13 , f ( e + ) is a strictly concave function and when 1 − ((1 − e − ) · e 2 − ) 13 <
Revised: Thus, when e − < e + < 1 − ((1 − e − ) · e 2 − ) 13 , f ( e + ) is a strictly concave function and when 1 − ((1 − e − ) · e 2 − ) 13 <

Original: e
Revised: e

Original: + < 1 − e − , f
Revised: + < 1 − e − , f

Original: ( e + ) is a strictly convex function.
Revised: ( e + ) is a strictly convex function.

Original: Since f (cid:48) ( e − ) > 0 and f (cid:48) (1 − e − ) = 0 , e 0 must locates in the point between e − and 1 − ((1 − e − ) ·
Revised: Since f (cid:48) ( e − ) > 0 and f (cid:48) (1 − e − ) = 0 , e 0 must locates in the point between e − and 1 − ((1 − e − ) ·

Original: e 2 − ) 13 which satisfy f (cid:48) ( e 0 ) = 0 .
Revised: e 2 − ) 13 which satisfy f (cid:48) ( e 0 ) = 0 .

Original: Thus when e − < e + < e 0 , f ( e + ) monotonically increases and when e 0 < e + < 1 − e − , f ( e + ) monotonically decreases.
Revised: Thus when e − < e + < e 0 , f ( e + ) monotonically increases and when e 0 < e + < 1 − e − , f ( e + ) monotonically decreases.

Original: Since f ( e − ) =
Revised: Since f ( e − ) =

Original: f (1 − e − ) = 0 .
Revised: f (1 − e − ) = 0 .

Original: We have f ( e + ) > 0 when e − < e + < 1 − e − .
Revised: We have f ( e + ) > 0 when e − < e + < 1 − e − .

Original: Proof done.
Revised: Proof done.

Original: We depict a ﬁgure in Figure 6 to better show the effect of down-sampling strategy.
Revised: We depict a ﬁgure in Figure 6 to better show the effect of down-sampling strategy.

Original: It can be seen the curves in the ﬁgure well support our proposition and proof.
Revised: It can be seen the curves in the ﬁgure well support our proposition and proof.

Original: When e + − e − is large, down-sampling strategy to make P ( (cid:101) Y = 1)
Revised: When e + − e − is large, down-sampling strategy to make P ( (cid:101) Y = 1)

Original: =
Revised: =

Original: P ( (cid:101) Y = 0) can well decrease the gap even we do not know the true value of e − and e + .
Revised: P ( (cid:101) Y = 0) can well decrease the gap even we do not know the true value of e − and e + .

Original: Thus down-sampling strategy can approximate the condition in Theorem 1, which explains why CE with ﬁxed encoder can also work well on instance-dependent label noise in
Revised: Thus down-sampling strategy can approximate the condition in Theorem 1, which explains why CE with ﬁxed encoder can also work well on instance-dependent label noise in

Original: Table 1.
Revised: Table 1.

Original: D M ORE D ISCUSSIONS AND E XPERIMENTS D. W HETHER OR NOT TO FIX THE ENCODER
Revised: D M ORE D ISCUSSIONS AND E XPERIMENTS D. W HETHER OR NOT TO FIX THE ENCODER

Original: Nodet et al.
Revised: Nodet et al.

Original: (2021) experimentally test many methods and losses based on the pre-trained SSL features and draw the conclusion that the encoder should not be ﬁxed during ﬁne-tuning.
Revised: (2021) experimentally test many methods and losses based on the pre-trained SSL features and draw the conclusion that the encoder should not be ﬁxed during ﬁne-tuning.

Original: We argue that this conclusion is not complete and neglects the specialty of each loss and the gap between SSL and SL varies for each dataset.
Revised: We argue that this conclusion is not complete and neglects the specialty of each loss and the gap between SSL and SL varies for each dataset.

Original: First, the robustness of CE in Equation (1) is based on the fact that
Revised: First, the robustness of CE in Equation (1) is based on the fact that

Original: CE is a calibrated and convex loss respect to the linear classiﬁer.
Revised: CE is a calibrated and convex loss respect to the linear classiﬁer.

Original: However, the losses tested by
Revised: However, the losses tested by

Original: Nodet et al.
Revised: Nodet et al.

Original: (2021) do not all satisfy this property when performing ﬁne-tuning.
Revised: (2021) do not all satisfy this property when performing ﬁne-tuning.

Original: For example, GCE
Revised: For example, GCE

Original: [Table]
Revised: [Table]

Original: (Zhang & Sabuncu, 2018) is not convex with respect to linear classiﬁer.
Revised: (Zhang & Sabuncu, 2018) is not convex with respect to linear classiﬁer.

Original: Second, if the gap between
Revised: Second, if the gap between

Original: SSL and SL is large, then the encoder should not be ﬁxed since supervised features can exhibit better performance.
Revised: SSL and SL is large, then the encoder should not be ﬁxed since supervised features can exhibit better performance.

Original: However, if the gap between SSL and SL is very small, it is guaranteed by Theorem 1 and Theorem 2 that CE on noisy dataset can approximates SL on clean dataset when encoder is ﬁxed.
Revised: However, if the gap between SSL and SL is very small, it is guaranteed by Theorem 1 and Theorem 2 that CE on noisy dataset can approximates SL on clean dataset when encoder is ﬁxed.

Original: The observation can be seen in Figure 3 (a).
Revised: The observation can be seen in Figure 3 (a).

Original: Thus, the above argument suggests that whether to ﬁx the encoder or not is determined by the gap between SSL and SL for a given dataset.
Revised: Thus, the above argument suggests that whether to ﬁx the encoder or not is determined by the gap between SSL and SL for a given dataset.

Original: D. P ROBLEM OF OTHER ROBUST LOSSES
Revised: D. P ROBLEM OF OTHER ROBUST LOSSES

Original: Many robust losses have been proposed in recent years.
Revised: Many robust losses have been proposed in recent years.

Original: For example, some losses (Natarajan et al., 2013; Liu & Guo, 2020) are also proven statistically robust which satisfy:
Revised: For example, some losses (Natarajan et al., 2013; Liu & Guo, 2020) are also proven statistically robust which satisfy:

Original: [Equation]
Revised: [Equation]

Original: (25) where (cid:96) is the robust loss and f , g in Equation (25) are simultaneously updated.
Revised: (25) where (cid:96) is the robust loss and f , g in Equation (25) are simultaneously updated.

Original: However, Equation (25) not only needs sufﬁcient training samples to achieve small learning error ε , but also does not consider the optimization difﬁculty of deep neural networks.
Revised: However, Equation (25) not only needs sufﬁcient training samples to achieve small learning error ε , but also does not consider the optimization difﬁculty of deep neural networks.

Original: Because of the great non-convexity of f , it is very hard to guarantee that f and g will converge to global optimal when noise rate varies.
Revised: Because of the great non-convexity of f , it is very hard to guarantee that f and g will converge to global optimal when noise rate varies.

Original: Thus the performance of (cid:96) in Equation (25) may suffer severe performance degradation under high noise rate.
Revised: Thus the performance of (cid:96) in Equation (25) may suffer severe performance degradation under high noise rate.

Original: However, for Equation (1), when f is ﬁxed, CE loss with respect to g is strictly convex which is very friendly for optimization.
Revised: However, for Equation (1), when f is ﬁxed, CE loss with respect to g is strictly convex which is very friendly for optimization.

Original: To further verify our analysis, we compare CE with SSL pre-trained encoder with some robust losses including SCE (Wang et al., 2019),
Revised: To further verify our analysis, we compare CE with SSL pre-trained encoder with some robust losses including SCE (Wang et al., 2019),

Original: APL (Ma et al., 2020) and Peer loss (Liu & Guo, 2020) on CIFAR10 with symmetric label noise.
Revised: APL (Ma et al., 2020) and Peer loss (Liu & Guo, 2020) on CIFAR10 with symmetric label noise.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: It can be observed that when noise rate is very high, the performance of these robust losses will drop signiﬁcantly.
Revised: It can be observed that when noise rate is very high, the performance of these robust losses will drop signiﬁcantly.

Original: However, when SSL pre-trained encoder is ﬁxed, CE loss with respect to linear classifer is strictly convex.
Revised: However, when SSL pre-trained encoder is ﬁxed, CE loss with respect to linear classifer is strictly convex.

Original: Thus the performance is more stable when noise rate increases.
Revised: Thus the performance is more stable when noise rate increases.

Original: D. T HE EFFECT OF DISTANCE MEASURE IN E QUATION (4)
Revised: D. T HE EFFECT OF DISTANCE MEASURE IN E QUATION (4)

Original: In this paper and experiment, we use l 2 norm to calculate the feature distance between SL features and square l 2 norm to calculate the distance between SSL features.
Revised: In this paper and experiment, we use l 2 norm to calculate the feature distance between SL features and square l 2 norm to calculate the distance between SSL features.

Original: This choice can lead to good performance from Theory 3 and Figure 5.
Revised: This choice can lead to good performance from Theory 3 and Figure 5.

Original: Practically, since structure regularization mainly captures the relations, different choice does not make a big effect on the performance.
Revised: Practically, since structure regularization mainly captures the relations, different choice does not make a big effect on the performance.

Original: We perform an experiment in Figure 7 which shows that the performance of both types are quite close.
Revised: We perform an experiment in Figure 7 which shows that the performance of both types are quite close.

Original: D. A BLATION STUDY
Revised: D. A BLATION STUDY

Original: In Figure 4, SSL training is to provide SSL features to regularize the output of linear classiﬁer g .
Revised: In Figure 4, SSL training is to provide SSL features to regularize the output of linear classiﬁer g .

Original: However, SSL training itself may have a positive effect on DNN.
Revised: However, SSL training itself may have a positive effect on DNN.

Original: To show the robustness mainly comes from the regularizer rather than SSL training, we perform an ablation study in Figure 8.
Revised: To show the robustness mainly comes from the regularizer rather than SSL training, we perform an ablation study in Figure 8.

Original: From the experiments, it is the regularizer that alleviates over-ﬁtting problem of DNN.
Revised: From the experiments, it is the regularizer that alleviates over-ﬁtting problem of DNN.

Original: D. V ISUALIZING FEATURES OF DNN TRAINED BY CE WHEN ENCODER IS NOT FIXED
Revised: D. V ISUALIZING FEATURES OF DNN TRAINED BY CE WHEN ENCODER IS NOT FIXED

Original: We visualize the features of DNN trained by CE when encoder is not ﬁxed.
Revised: We visualize the features of DNN trained by CE when encoder is not ﬁxed.

Original: Figure 9 shows the Tsne visualization of features before linear classiﬁer under different noise rate.
Revised: Figure 9 shows the Tsne visualization of features before linear classiﬁer under different noise rate.

Original: It can be seen, as analyzed
Revised: It can be seen, as analyzed

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: in the paper, due to non-convexity and great over-ﬁtting capability of f (encoder), the features of f ( X ) have already been contaminated by the label noise before linear classiﬁer, thus CE on vanilla ( X, (cid:101) Y ) is not robust.
Revised: in the paper, due to non-convexity and great over-ﬁtting capability of f (encoder), the features of f ( X ) have already been contaminated by the label noise before linear classiﬁer, thus CE on vanilla ( X, (cid:101) Y ) is not robust.

Original: E D ETAILED SETTING OF EXPERIMENTS
Revised: D. T HE EFFECT OF DIFFERENT SSL- PRETRAINED METHODS

Revised: Our analyses and experiments are not restricted to any speciﬁc SSL method. Experimentally, other SSL methods are also adoptable to pre-train SSL encoders.

Revised: In Figure 3, SimCLR (Chen et al., 2020) is adopted to pre-train SSL encoder.

Revised: For a comparison, we pre-train a encoder with Moco (He et al., 2020) on CIFAR10 and ﬁne-tune linear classiﬁer on noisy labels in Table 3.

Revised: [Table]

Revised: It can be observed that different SSL methods have very similar results, which further veriﬁes our

Revised: Theorems in the paper.

Revised: D. E XPERIMENTS ON H UMAN - ANNOTATED N OISY D ATASET

Revised: We perform experiments on CIFAR100-N (A challenging human-annotated noisy dataset) (Wei et al., 2021) with structure regularization.

Revised: From Table 4, it can be observed that by using SSL features, Structure Regularization can outperform many benchmark methods on real-world human annotated noisy dataset.

Revised: [Table]

Revised: E D ETAILED SETTING OF EXPERIMENTS

Original: Datasets: We use DogCat, CIFAR10 and CIFAR100 for experiments.
Revised: Datasets: We use DogCat, CIFAR10 and CIFAR100 for experiments.

Original: DogCat has 25000 images.
Revised: DogCat has 25000 images.

Original: We randomly choose 24000 images for training and 1000 images for testing.
Revised: We randomly choose 24000 images for training and 1000 images for testing.

Original: For CIFAR10 and
Revised: For CIFAR10 and

Original: CIFAR100, we follow standard setting that use 50000 images for training and 10000 images for testing.
Revised: CIFAR100, we follow standard setting that use 50000 images for training and 10000 images for testing.

Original: Setting in Section 3: SimCLR is deployed for SSL pre-training with ResNet50 for DogCat and
Revised: Setting in Section 3: SimCLR is deployed for SSL pre-training with ResNet50 for DogCat and

Original: ResNet34 for CIFAR10 and CIFAR100.
Revised: ResNet34 for CIFAR10 and CIFAR100.

Original: Each model is pre-trained by 1000 epochs with Adam optimizer (lr = 1e-3) and batch-size is set to be 512.
Revised: Each model is pre-trained by 1000 epochs with Adam optimizer (lr = 1e-3) and batch-size is set to be 512.

Original: During ﬁne-tuning, we ﬁx the encoder and only ﬁne-tune the linear classiﬁer on noisy dataset with Adam (lr = 1e-3) for 100 epochs and batch-size is set to be 256.
Revised: During ﬁne-tuning, we ﬁx the encoder and only ﬁne-tune the linear classiﬁer on noisy dataset with Adam (lr = 1e-3) for 100 epochs and batch-size is set to be 256.

Original: Setting in Section 4: The basic hyper-parameters are identical to Section 3, except when we train from scratch (random initialization), learning rate is set to be 0.1 at initial state and is decayed by 0.1 at 50 epochs.
Revised: Setting in Section 4: The basic hyper-parameters are identical to Section 3, except when we train from scratch (random initialization), learning rate is set to be 0.1 at initial state and is decayed by 0.1 at 50 epochs.

Original: In Equation (7), we use MSE loss for measuring the relations between SL features and SSL features.
Revised: In Equation (7), we use MSE loss for measuring the relations between SL features and SSL features.

Original: However, since MSE loss may cause gradient exploration when prediction is far from ground-truth, we use smooth l 1 loss instead.
Revised: However, since MSE loss may cause gradient exploration when prediction is far from ground-truth, we use smooth l 1 loss instead.

Original: Smooth l 1 loss is an enhanced version of MSE loss.
Revised: Smooth l 1 loss is an enhanced version of MSE loss.

Original: When prediction is not very far from ground-truth, smooth l 1 loss is MSE, and MAE when prediction is far.
Revised: When prediction is not very far from ground-truth, smooth l 1 loss is MSE, and MAE when prediction is far.

Original: The code with running guideline has already been attached in the supplementary material.
Revised: The code with running guideline has already been attached in the supplementary material.

Revised: F G ENERALIZE T HEOREM 1 TO MULTI - CLASS CLASSIFICATION

Revised: Theorem 5 For K-class classiﬁcation, let g

Revised: [Equation]

Revised: D [ CE ( g ( f ( X )) , (cid:101) Y )] .

Revised: Then if symmetric noise ratio (cid:15) < K − 1 K , for each sample x i , we have

Revised: [Equation]

Revised: (26) where f is ﬁxed encoder and g is the linear classiﬁer, g ( f ( · )) denotes the network output after softmax layer.

Revised: proof: Following the proof for Theorem 1, suppose g Bayesand g Bayesare Bayes optimal classiﬁers under P ( f ( X ) , Y ) and P ( f ( X ) , (cid:101) Y ) , respectively.

Revised: We aim to show g Bayesand g Bayesare consistent.

Revised: I.e., the decision boundaries are the same.

Revised: Denote X i =

Revised: X | Y

Revised: = i .

Revised: Deﬁne p

Revised: i ( x ) to be the distribution of f ( X i ) .

Revised: According to the bayes rules:

Revised: [Equation]

Revised: Let X be the space of f ( X ) :

Revised: X =

Revised: X

Revised: [Equation]

Revised: [Equation]

Revised: ∪

Revised: [Equation]

Revised: K (28) where X i = { x ∈ f ( X ) | P ( Y = i | x ) > P ( Y =

Revised: j | x ) , i (cid:54) = j } .

Revised: Let p (cid:15)i ( x ) be the distribution of samples whose observed labels are i under symmetric label noise rate (cid:15) .

Revised: We have:

Revised: [Equation]

Revised: where m i is the normalization term to make distribution valid.

Revised: Speciﬁcally, m i =

Revised: P ( Y = i ) · (1 − (cid:15) )

Revised: + (cid:80) j (cid:54) =

Revised: i P ( Y = j ) · (cid:15)K − 1 .

Revised: Under label noise, P ( (cid:101) Y = i ) =

Revised: P ( Y = i ) · (1 − (cid:15) )

Revised: + (cid:80) j (cid:54) =

Revised: i P ( Y = j ) · (cid:15)K − 1 = 1 m i .

Revised: According to the bayes rules, we have:

Revised: [Equation]

Revised: Let X = X (cid:15) 1 ∪ X (cid:15) 2 · · · ∪ X (cid:15)K , where X (cid:15)i = { x ∈ f ( X ) | P ( (cid:101) Y = i | x ) > P ( (cid:101) Y =

Revised: j | x ) , i (cid:54) = j } .

Revised: To prove g Bayesand g Bayesare consistent, it is equivalent to show X i =

Revised: X (cid:15)i when (cid:15) <

Revised: K − 1 K .

Revised: Without loss of generality, we prove X 1 = X (cid:15) 1 when (cid:15) <

Revised: K − 1 K .

Revised: When j (cid:54) = 1 , we have:

Revised: [Equation]

Revised: normalization (31)

Revised: Since X 1 = { x ∈ f ( X ) | P ( Y = 1 | x ) > P ( Y = j | x ) , j (cid:54) = 1 } , X 1 = X (cid:15) 1 if 1 − (cid:15) − (cid:15)K − 1 > 0 , i.e., (cid:15) <

Revised: K − 1 K .

Revised: Proof Done.

Revised: G I LLUSTRATING B AYES CLASSIFIER IS NOT CONSISTENT UNDER ASYMMETRIC AND INSTANCE LABEL NOISE

Revised: Asymmetric label noise: We give an example on K-class classiﬁcation (K=3).

Revised: Following the definition and notation in Section F, suppose the asymmetric label noise ratio is (cid:15) , which is generated between two adjacent label ( 1 ⇔ 2 , 2 ⇔ 3 , 3 ⇔ 1 ), we have:

Revised: [Equation]

Revised: normalization

Revised: [Equation]

Revised: [Equation]

Revised: normalization (33)

Revised: Since X 1 = { x ∈ f ( X ) | P ( Y = 1 | x ) > P ( Y = j | x ) , j (cid:54) = 1 } and X (cid:15) 1 = { x ∈ f ( X ) | P ( (cid:101)

Original: {+Y = 
+}
Revised: {+Y = 
+}

Revised: | x ) >

Revised: P ( (cid:101) Y =

Revised: j | x ) , j (cid:54) = 1 } , it is easy to verify X 1 =

Revised: X (cid:15) 1 only when (cid:15)

Revised: = 0 .

Revised: Thus asymmetric label noise does not induce consistent Bayes classiﬁer.

Revised: We also provide an experiment on CIFAR10 to further verify our analysis in Table 5.

Revised: It can be seen that for asymmetric label noise, there is only marginal improvement compared to symmetric label noise in Figure 3.

Revised: [Table]

Revised: Instance-dependent label noise : Instance-dependent label noise behaves like symmetric label noise, i.e., the noisy labels may exist in all the classes.

Revised: However, the noise ratio for each class is not the same and dependent on the features.

Revised: From Section F, when noise ratio for each class is not the same, the Bayes classiﬁer is not consistent.

Revised: However, from Table 1 and Section C we theoretically and experimentally show that simple down-sampling strategy can approximate condition in Theorem 1 and Theorem 5 which is helpful for instance level label noise.