Original: I S THE L ABEL T RUSTFUL : T RAINING B ETTER D EEP L EARNING M ODEL VIA U NCERTAINTY M INING N ET
Revised: I S THE L ABEL T RUSTFUL : T RAINING B ETTER D EEP L EARNING M ODEL VIA U NCERTAINTY M INING N ET

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work, we consider a new problem of training deep neural network on partially labeled data with label noise.
Revised: In this work, we consider a new problem of training deep neural network on partially labeled data with label noise.

Original: As far as we know, there have been very few efforts to tackle such problems.
Revised: As far as we know, there have been very few efforts to tackle such problems.

Original: We present a novel end-to-end deep generative pipeline for improving classiﬁer performance when dealing with such data problems.
Revised: We present a novel end-to-end deep generative pipeline for improving classiﬁer performance when dealing with such data problems.

Original: We call it Uncertainty Mining Net (UMN).
Revised: We call it Uncertainty Mining Net (UMN).

Original: During the training stage, we utilize all the available data (labeled and unlabeled) to train the classiﬁer via a semi-supervised generative framework.
Revised: During the training stage, we utilize all the available data (labeled and unlabeled) to train the classiﬁer via a semi-supervised generative framework.

Original: During training, UMN estimates the uncertainly of the labels’ to focus on clean data for learning.
Revised: During training, UMN estimates the uncertainly of the labels’ to focus on clean data for learning.

Original: More precisely, UMN applies the samplewise label uncertainty estimation scheme.
Revised: More precisely, UMN applies the samplewise label uncertainty estimation scheme.

Original: Extensive experiments and comparisons against state-of-the-art methods on several popular benchmark datasets demonstrate that UMN can reduce the effects of label noise and signiﬁcantly improve classiﬁer performance.
Revised: Extensive experiments and comparisons against state-of-the-art methods on several popular benchmark datasets demonstrate that UMN can reduce the effects of label noise and signiﬁcantly improve classiﬁer performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Learning (DL), to learn powerful representations, it usually requires a large amount of training data.
Revised: Deep Learning (DL), to learn powerful representations, it usually requires a large amount of training data.

Original: However, for many real world problems, it is not always possible to obtain sufﬁciently large training data.
Revised: However, for many real world problems, it is not always possible to obtain sufﬁciently large training data.

Original: What we can usually get is limited training data with corrupted labels which heavily affect the model performance.
Revised: What we can usually get is limited training data with corrupted labels which heavily affect the model performance.

Original: Although acquiring large data is not hard, considering the information explosion on the internet, accurate labeling is usually an expensive and error-prone task which involves humans’ interaction, especially experts with knowledge in the speciﬁc ﬁeld.
Revised: Although acquiring large data is not hard, considering the information explosion on the internet, accurate labeling is usually an expensive and error-prone task which involves humans’ interaction, especially experts with knowledge in the speciﬁc ﬁeld.

Original: Most of the time, we have to build DL models using limited training data with corrupted data labels.
Revised: Most of the time, we have to build DL models using limited training data with corrupted data labels.

Original: It becomes very challenging to apply current popular deep learning frameworks to solve this problem.
Revised: It becomes very challenging to apply current popular deep learning frameworks to solve this problem.

Original: Training a deep learning model via noisily labeled data is a challenging task (Patrini et al., 2017;
Revised: Training a deep learning model via noisily labeled data is a challenging task (Patrini et al., 2017;

Original: Li et al., 2017; Ding et al., 2018; Menon et al., 2015; Goldberger & Ben-Reuven, 2016; Jiang et al., 2017).
Revised: Li et al., 2017; Ding et al., 2018; Menon et al., 2015; Goldberger & Ben-Reuven, 2016; Jiang et al., 2017).

Original: Most of current explorations for dealing with such problems usually require large amount of labeled data as a prerequisite.
Revised: Most of current explorations for dealing with such problems usually require large amount of labeled data as a prerequisite.

Original: It means that in these works, the designed pipelines and evaluations are based on having enough labeled training data, though they may include label noise.
Revised: It means that in these works, the designed pipelines and evaluations are based on having enough labeled training data, though they may include label noise.

Original: These approaches will suffer from the limited size of labeled training data as well as the label noise.
Revised: These approaches will suffer from the limited size of labeled training data as well as the label noise.

Original: As we know, the hierarchical representation learned by deep learning models mainly beneﬁts from the amount of data.
Revised: As we know, the hierarchical representation learned by deep learning models mainly beneﬁts from the amount of data.

Original: The deep learning model can easily overﬁt to the mislabeled samples especially when the data size is small (Tarvainen & Valpola, 2017; Ren et al., 2018).
Revised: The deep learning model can easily overﬁt to the mislabeled samples especially when the data size is small (Tarvainen & Valpola, 2017; Ren et al., 2018).

Original: Unfortunately, there is no general solutions to solve the problem of training on limited annotated data with label noise.
Revised: Unfortunately, there is no general solutions to solve the problem of training on limited annotated data with label noise.

Original: If we directly train the deep learning model only with the labeled data, the small portion of labeled training data will limit the learning capability.
Revised: If we directly train the deep learning model only with the labeled data, the small portion of labeled training data will limit the learning capability.

Original: Meanwhile, the label noise makes it very challenging to utilize the labeled data directly.
Revised: Meanwhile, the label noise makes it very challenging to utilize the labeled data directly.

Original: In this work, our framework can handle these problems simultaneously.
Revised: In this work, our framework can handle these problems simultaneously.

Original: UMN includes two major components.
Revised: UMN includes two major components.

Original: The ﬁrst part is used to learn a latent feature representation via a generative framework (VAE) using all the training data.
Revised: The ﬁrst part is used to learn a latent feature representation via a generative framework (VAE) using all the training data.

Original: Then the learned embedding is applied as the input to its subsequent semi-supervised learning component by conditional variational autoencoder to train the classiﬁer.
Revised: Then the learned embedding is applied as the input to its subsequent semi-supervised learning component by conditional variational autoencoder to train the classiﬁer.

Original: To handle the label noise, we integrate label uncertainty estimation module where the reliable data contributes more to the model training and noisy data contributes less.
Revised: To handle the label noise, we integrate label uncertainty estimation module where the reliable data contributes more to the model training and noisy data contributes less.

Original: Though recent works (Hataya & Nakayama, 2019; Ren et al., 2018) discuss such problems, they usually assume to have a separate small clean dataset to begin with.
Revised: Though recent works (Hataya & Nakayama, 2019; Ren et al., 2018) discuss such problems, they usually assume to have a separate small clean dataset to begin with.

Original: It may not be possible to have an independent clean dataset in real scenarios.
Revised: It may not be possible to have an independent clean dataset in real scenarios.

Original: Current supervised robust learning approaches cannot be directly adopted to our case since the labeled data is small and not reliable.
Revised: Current supervised robust learning approaches cannot be directly adopted to our case since the labeled data is small and not reliable.

Original: The limited labeled training data makes the supervised model overﬁt very easily to the small size of noisy data.
Revised: The limited labeled training data makes the supervised model overﬁt very easily to the small size of noisy data.

Original: This will heavily hurt the model performance.
Revised: This will heavily hurt the model performance.

Original: The limited data labels and noise can also affect other related state-of-the-art semi-supervised learning approaches.
Revised: The limited data labels and noise can also affect other related state-of-the-art semi-supervised learning approaches.

Original: We will present the comparisons with these approaches in the experiment section.
Revised: We will present the comparisons with these approaches in the experiment section.

Original: In this paper, our major contributions can be summarized into the following aspects: First, our studied problem is very general in real scenarios: small mislabeled datasets with large amount of unlabeled data, and there are very few existing approaches to target this kind of problems.
Revised: In this paper, our major contributions can be summarized into the following aspects: First, our studied problem is very general in real scenarios: small mislabeled datasets with large amount of unlabeled data, and there are very few existing approaches to target this kind of problems.

Original: Second, instead of treating the label noise as the preknowledge, we explicitly use the deep generative architecture to model the noisy labels at the sample level precisely.
Revised: Second, instead of treating the label noise as the preknowledge, we explicitly use the deep generative architecture to model the noisy labels at the sample level precisely.

Original: This has been validated by the improved experimental results as well as the theoretical analysis.
Revised: This has been validated by the improved experimental results as well as the theoretical analysis.

Original: Third, we show how to use the moving average model to estimate the sample-wise uncertainty in labels.
Revised: Third, we show how to use the moving average model to estimate the sample-wise uncertainty in labels.

Original: We have explained it in the theory analysis section and validated this assumption in the experiments.
Revised: We have explained it in the theory analysis section and validated this assumption in the experiments.

Original: Furthermore, our experiments demonstrate that UMN can also help identify these mis-labeled data without any prior knowledge.
Revised: Furthermore, our experiments demonstrate that UMN can also help identify these mis-labeled data without any prior knowledge.

Original: Finally, we build one end-to-end deep learning pipeline to train against the noisy labeled data.
Revised: Finally, we build one end-to-end deep learning pipeline to train against the noisy labeled data.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Along with the popularity of deep neural network, the problem of training deep neural networks with noisy label data has started to draw our attention.
Revised: Along with the popularity of deep neural network, the problem of training deep neural networks with noisy label data has started to draw our attention.

Original: Natarajan et al.
Revised: Natarajan et al.

Original: (Natarajan et al., 2013) propose the unbiased estimator of the surrogate loss function and calculate theoretical bounds for empirical risk optimization.
Revised: (Natarajan et al., 2013) propose the unbiased estimator of the surrogate loss function and calculate theoretical bounds for empirical risk optimization.

Original: Modeling the consistency as the regularization for deep neural network is another route for robust learning with noisy labeled data.
Revised: Modeling the consistency as the regularization for deep neural network is another route for robust learning with noisy labeled data.

Original: Reed et al.
Revised: Reed et al.

Original: (Reed et al., 2015) design a robust loss to model the prediction consistency.
Revised: (Reed et al., 2015) design a robust loss to model the prediction consistency.

Original: In (Li et al., 2017), the authors propose a knowledge distillation framework where they train an auxiliary model on a small set of clean data samples and linearly combine its predictions with the observed labels to form the new targets to train.
Revised: In (Li et al., 2017), the authors propose a knowledge distillation framework where they train an auxiliary model on a small set of clean data samples and linearly combine its predictions with the observed labels to form the new targets to train.

Original: In (Ding et al., 2018), a two-stage approach is proposed.
Revised: In (Ding et al., 2018), a two-stage approach is proposed.

Original: First a DNN model is trained on the noisy data and used to ﬁlter out these noisy samples.
Revised: First a DNN model is trained on the noisy data and used to ﬁlter out these noisy samples.

Original: Then another model is trained on the ﬁltered dataset using a semi-supervised approach by treating these ﬁltered out samples as unlabeled.
Revised: Then another model is trained on the ﬁltered dataset using a semi-supervised approach by treating these ﬁltered out samples as unlabeled.

Original: Menon et al.
Revised: Menon et al.

Original: model the corruption process of a dataset by learning the class-probability estimator (Menon et al., 2015).
Revised: model the corruption process of a dataset by learning the class-probability estimator (Menon et al., 2015).

Original: Another way of regularization for training on noisy labels is to estimate the class conditional corruption ratio.
Revised: Another way of regularization for training on noisy labels is to estimate the class conditional corruption ratio.

Original: Goldberger et al.
Revised: Goldberger et al.

Original: (Goldberger & Ben-Reuven, 2016) propose a softmax layer along with the classiﬁer to predict the class conditional corruption ratio.
Revised: (Goldberger & Ben-Reuven, 2016) propose a softmax layer along with the classiﬁer to predict the class conditional corruption ratio.

Original: In (Jiang et al., 2017), they propose MentorNet which learns a data driven dynamic curriculum to be followed by the StudentNet.
Revised: In (Jiang et al., 2017), they propose MentorNet which learns a data driven dynamic curriculum to be followed by the StudentNet.

Original: Semi-supervised learning with few labeled samples is also a well studied area.
Revised: Semi-supervised learning with few labeled samples is also a well studied area.

Original: Some of the closely related works are (Gordon & Hernández-Lobato, 2017), (Kingma et al., 2014), (Tarvainen & Valpola, 2017).
Revised: Some of the closely related works are (Gordon & Hernández-Lobato, 2017), (Kingma et al., 2014), (Tarvainen & Valpola, 2017).

Original: Kingma et al (Kingma et al., 2014) propose a stacked deep generative semi-supervised model for training on partially labeled datasets.
Revised: Kingma et al (Kingma et al., 2014) propose a stacked deep generative semi-supervised model for training on partially labeled datasets.

Original: They ﬁrst train a variational autoencoder on the labeled data, then stack on top of it a conditional variational autoencoder and continue training in semi-supervised fashion to get a robust classiﬁer.
Revised: They ﬁrst train a variational autoencoder on the labeled data, then stack on top of it a conditional variational autoencoder and continue training in semi-supervised fashion to get a robust classiﬁer.

Original: In (Tarvainen & Valpola, 2017), the authors train a classiﬁer in semi-supervised way using temporal ensemble approach.
Revised: In (Tarvainen & Valpola, 2017), the authors train a classiﬁer in semi-supervised way using temporal ensemble approach.

Original: They train a student network and maintain a teacher network as the exponential moving average of the student.
Revised: They train a student network and maintain a teacher network as the exponential moving average of the student.

Original: They use classiﬁcation loss for labeled data and enforce consistency loss between teacher and student for unlabeled data.
Revised: They use classiﬁcation loss for labeled data and enforce consistency loss between teacher and student for unlabeled data.

Original: There have been limited works dealing with bi-quality data (Hataya & Nakayama, 2019), (Langevin et al., 2018).
Revised: There have been limited works dealing with bi-quality data (Hataya & Nakayama, 2019), (Langevin et al., 2018).

Original: Bi-quality data is deﬁned in (Hataya & Nakayama, 2019) as datasets with few labeled samples where the labels are potentially corrupted.
Revised: Bi-quality data is deﬁned in (Hataya & Nakayama, 2019) as datasets with few labeled samples where the labels are potentially corrupted.

Original: In (Langevin et al., 2018), the authors discuss to use the deep generative semi-supervised model to model noisy labels.
Revised: In (Langevin et al., 2018), the authors discuss to use the deep generative semi-supervised model to model noisy labels.

Original: They assume that the labels are also randomly corrupted with a probability ρ .
Revised: They assume that the labels are also randomly corrupted with a probability ρ .

Original: Their idea relies on the predeﬁned ρ to train the classiﬁer.
Revised: Their idea relies on the predeﬁned ρ to train the classiﬁer.

Original: However, the corruption probability ρ is not known as a prior in reality.
Revised: However, the corruption probability ρ is not known as a prior in reality.

Original: Compared to these work, we present a new architecture for training a robust classiﬁer with dataset including noisy labels.
Revised: Compared to these work, we present a new architecture for training a robust classiﬁer with dataset including noisy labels.

Original: We assume that there is no preknowledge of the label corruption rate and we do not have access to a separate clean dataset which is more close to the real scenario.
Revised: We assume that there is no preknowledge of the label corruption rate and we do not have access to a separate clean dataset which is more close to the real scenario.

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: q q ’
Revised: q q ’

Original: [Figure]
Revised: [Figure]

Original: ReconCost R e c o n
Revised: ReconCost R e c o n

Original: C o s t
Revised: C o s t

Original: +
Revised: +

Original: K L c o s t
Revised: K L c o s t

Original: P r e d i c t i o n P r e d i c t i o n h h ’
Revised: P r e d i c t i o n P r e d i c t i o n h h ’

Original: [Figure]
Revised: [Figure]

Original:  M ETHOD
Revised:  M ETHOD

Original: The goal of our work is to learn a better model with the limited labeled data including the noise.
Revised: The goal of our work is to learn a better model with the limited labeled data including the noise.

Original: Since the labeled data only occupies a small portion of the training data, the labeled data alone is not sufﬁcient to train a good model.
Revised: Since the labeled data only occupies a small portion of the training data, the labeled data alone is not sufﬁcient to train a good model.

Original: At the same time, we also need to solve the label noise problem.
Revised: At the same time, we also need to solve the label noise problem.

Original: To deal with these issues, our pipeline consists of two major components.
Revised: To deal with these issues, our pipeline consists of two major components.

Original: In the ﬁrst part, we apply unsupervised learning scheme to learn a latent feature representation via all the available training data.
Revised: In the ﬁrst part, we apply unsupervised learning scheme to learn a latent feature representation via all the available training data.

Original: Then we utilize the label via semi-supervised learning pipeline to train the classiﬁer.
Revised: Then we utilize the label via semi-supervised learning pipeline to train the classiﬁer.

Original: At the same time, we incorporate the label uncertainty estimation to reduce the inﬂuences coming from these mislabeled data.
Revised: At the same time, we incorporate the label uncertainty estimation to reduce the inﬂuences coming from these mislabeled data.

Original: We estimate the uncertainty of the given sample from the prediction of the updating model and its guider model (exponential moving average of the updating model), part of this idea is inspired by work (Polyak & Juditsky, 1992).
Revised: We estimate the uncertainty of the given sample from the prediction of the updating model and its guider model (exponential moving average of the updating model), part of this idea is inspired by work (Polyak & Juditsky, 1992).

Original: The estimated uncertainty is then used to assign weights to the training data samples.
Revised: The estimated uncertainty is then used to assign weights to the training data samples.

Original: This is an end-to-end framework.
Revised: This is an end-to-end framework.

Original: We call it Uncertainty
Revised: We call it Uncertainty

Original: Mining Nets(UMN).
Revised: Mining Nets(UMN).

Original: In the following subsection, we will introduce UMN in more details.
Revised: In the following subsection, we will introduce UMN in more details.

Original: 3. L ATENT - FEATURE L EARNING
Revised: 3. L ATENT - FEATURE L EARNING

Original: In this work, we represent the given data as the set of ( X, Y ) =
Revised: In this work, we represent the given data as the set of ( X, Y ) =

Original: { ( x 1 , y 1 ) , . . .
Revised: { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } where x i is the observed data sample
Revised: , ( x N , y N ) } where x i is the observed data sample

Original: and y i is its corresponding label which may be potentially corrupted.
Revised: and y i is its corresponding label which may be potentially corrupted.

Original: If x i is unlabeled, then y i becomes empty.
Revised: If x i is unlabeled, then y i becomes empty.

Original: To be concise, we omit the index i in the rest of the paper.
Revised: To be concise, we omit the index i in the rest of the paper.

Original: Motivated by the recent success of generative models in semi-supervised learning related applications, ﬁrstly, we apply the variational autoencoder (VAE) on all the training data.
Revised: Motivated by the recent success of generative models in semi-supervised learning related applications, ﬁrstly, we apply the variational autoencoder (VAE) on all the training data.

Original: VAE consists of two modules: an Encoder that maps the variables x to the latent variables z to approximate the prior distribution of p ( z ) and a Generator p θ ( x | z ) that samples the the input variables x given the latent variables z .
Revised: VAE consists of two modules: an Encoder that maps the variables x to the latent variables z to approximate the prior distribution of p ( z ) and a Generator p θ ( x | z ) that samples the the input variables x given the latent variables z .

Original: It can be formulated as
Revised: It can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where f ( x ; z, θ ) is the likelihood function and N indicates the Gaussian distribution.
Revised: where f ( x ; z, θ ) is the likelihood function and N indicates the Gaussian distribution.

Original: θs are network parameters.
Revised: θs are network parameters.

Original: The goal of VAE is to maximize the evidence variational lower bound (ELBO) of p θ ( x ) as log p θ ( x ) (cid:62) E q φ ( z | x ) log p θ ( x | z )
Revised: The goal of VAE is to maximize the evidence variational lower bound (ELBO) of p θ ( x ) as log p θ ( x ) (cid:62) E q φ ( z | x ) log p θ ( x | z )

Original: − D KL ( q φ ( z | x ) || p ( x )) .
Revised: − D KL ( q φ ( z | x ) || p ( x )) .

Original: The learned latent variables obtained from VAE can be used as the feature representation to train the classiﬁer.
Revised: The learned latent variables obtained from VAE can be used as the feature representation to train the classiﬁer.

Original: Based on this, we can fully utilize the knowledge from the training data without the constraint of the labeled data size limitation.
Revised: Based on this, we can fully utilize the knowledge from the training data without the constraint of the labeled data size limitation.

Original: The low-dimensional embedding z can well cover the distribution of the input data x .
Revised: The low-dimensional embedding z can well cover the distribution of the input data x .

Original: Our hypothesis is that the unlabeled data helps training robustly against the noisy labels via the deep generative modeling.
Revised: Our hypothesis is that the unlabeled data helps training robustly against the noisy labels via the deep generative modeling.

Original: We will see this in the next subsection.
Revised: We will see this in the next subsection.

Original: 3. S EMI - SUPERVISED L EARNING
Revised: 3. S EMI - SUPERVISED L EARNING

Original: After building the unsupervised learning framework via VAE, we apply the label information to improve the learning capability of the neural network.
Revised: After building the unsupervised learning framework via VAE, we apply the label information to improve the learning capability of the neural network.

Original: Instead of using the raw data x , we directly use the latent vector z a obtained from VAE of latent-feature learning part along with the pre-deﬁned labels.
Revised: Instead of using the raw data x , we directly use the latent vector z a obtained from VAE of latent-feature learning part along with the pre-deﬁned labels.

Original: We follow the idea of conditional generative model.
Revised: We follow the idea of conditional generative model.

Original: Our generative process can be formulated as,
Revised: Our generative process can be formulated as,

Original: [Equation]
Revised: [Equation]

Original: where M is the multinomial distribution.
Revised: where M is the multinomial distribution.

Original: y indicates the true label.
Revised: y indicates the true label.

Original: Similar as z a, z b is jointly generated from z a and y .
Revised: z a is jointly generated from z b and y and z b is the latent representation of the input.

Original: In this work, we treat y as the latent variables.
Revised: In this work, we treat y as the latent variables.

Original: We use ˆ y to denote the observed labels.
Revised: We use ˆ y to denote the observed labels.

Original: To more accurately capture the reliability of labels, we estimate each labeled sample uncertainty via p θ (ˆ y | y, z a ) .
Revised: To more accurately capture the reliability of labels, we estimate each labeled sample uncertainty via p θ (ˆ y | y, z a ) .

Original: the posterior distribution can be factorised as
Revised: the posterior distribution can be factorised as

Original: [Equation]
Revised: [Equation]

Original: Following this, the Evidence Lower Bound (ELBO) can be derived as follows:
Revised: Following this, the Evidence Lower Bound (ELBO) can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( y | z a ) , q φ ( z a | x ) , q φ ( z b | z a , y ) , p θ ( z a | z b , y ) , p θ ( x | z a ) indicate the classiﬁer, encoder, conditional encoder, conditional decoder and decoder functions respectively.
Revised: where q φ ( y | z a ) , q φ ( z a | x ) , q φ ( z b | z a , y ) , p θ ( z a | z b , y ) , p θ ( x | z a ) indicate the classiﬁer, encoder, conditional encoder, conditional decoder and decoder functions respectively.

Original: L indicates labeled data.
Revised: L indicates labeled data.

Original: U means unlabeled data and C denotes the set of used classes.
Revised: U means unlabeled data and C denotes the set of used classes.

Original: The last term can be treated as labeled loss term as the supervised portion of Eq.
Revised: The last term can be treated as labeled loss term as the supervised portion of Eq.

Original: For a more detailed derivation of the ELBO, please refer to the Appendix.
Revised: For a more detailed derivation of the ELBO, please refer to the Appendix.

Original: As indicated in Eq.
Revised: As indicated in Eq.

Original: 5, except the last term, all other terms in the equation are calculated over all training data samples.
Revised: 5, except the last term, all other terms in the equation are calculated over all training data samples.

Original: The last term serves as the supervised factor for semi-supervised learning where the observed labels may be incorrect.
Revised: The last term serves as the supervised factor for semi-supervised learning where the observed labels may be incorrect.

Original: It uses an uncertainty related function p θ (ˆ y | y k , z a ) to modulate the sample weights during the gradient back-propagation.
Revised: It uses an uncertainty related function p θ (ˆ y | y k , z a ) to modulate the sample weights during the gradient back-propagation.

Original: As indicated in Fig.
Revised: As indicated in Fig.

Original: 1, p θ (ˆ y | y k , z a ) is approximated based on the sample-wise uncertainty (cid:15) .
Revised: 1, p θ (ˆ y | y k , z a ) is approximated based on the sample-wise uncertainty (cid:15) .

Original: More details are given in the following.
Revised: More details are given in the following.

Original: For simplicity, we can also represent the generative process of the observed ˆ y by Fig.
Revised: For simplicity, we can also represent the generative process of the observed ˆ y by Fig.

Original: One plausible way to estimate the uncertainty of the sample label is to model the class conditional corruption ratio p (ˆ y | y ) , which is a mean ﬁeld approximation.
Revised: One plausible way to estimate the uncertainty of the sample label is to model the class conditional corruption ratio p (ˆ y | y ) , which is a mean ﬁeld approximation.

Original: As discussed in (Langevin et al., 2018), the class conditional probability can be computed as
Revised: As discussed in (Langevin et al., 2018), the class conditional probability can be computed as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: and the labeled loss term can be formulated as
Revised: and the labeled loss term can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where f ( (cid:15) )
Revised: where f ( (cid:15) )

Original: =
Revised: =

Original: log
Revised: log

Original: [ ( C − 1)(1 − (cid:15) )
Revised: [ ( C − 1)(1 − (cid:15) )

Original: (cid:15) ] is a function with constant (cid:15) .
Revised: (cid:15) ] is a function with constant (cid:15) .

Original: From Eq.
Revised: From Eq.

Original: 7, we can see that it assigns the same weight for all samples belonging to the same category, i.e., all labeled terms contribute equally to the gradient back propagated regardless of their label correctness as
Revised: 7, we can see that it assigns the same weight for all samples belonging to the same category, i.e., all labeled terms contribute equally to the gradient back propagated regardless of their label correctness as

Original: [Equation]
Revised: [Equation]

Original: Although this idea is simple and easy to follow, the assumption may not be the case in the real scenario.
Revised: Although this idea is simple and easy to follow, the assumption may not be the case in the real scenario.

Original: In general, the value of (cid:15) which is the class conditional corruption rate, remains unknown beforehand.
Revised: In general, the value of (cid:15) which is the class conditional corruption rate, remains unknown beforehand.

Original: Another issue is that it’s not accurate to apply the same categorical corruption ratio to all data samples of a given class.
Revised: Another issue is that it’s not accurate to apply the same categorical corruption ratio to all data samples of a given class.

Original: Our approach solves these problems by estimating sample-wise uncertainty during the training stage.
Revised: Our approach solves these problems by estimating sample-wise uncertainty during the training stage.

Original: We replace the class conditional (cid:15) in Eq.
Revised: We replace the class conditional (cid:15) in Eq.

Original: 9 with a point-wise estimation of the label uncertainty (cid:15) ( z a ) , where we follow the generative process of Eq.
Revised: 9 with a point-wise estimation of the label uncertainty (cid:15) ( z a ) , where we follow the generative process of Eq.

Original: The uncertainty of label is hence (cid:15) ( z a ) .
Revised: The uncertainty of label is hence (cid:15) ( z a ) .

Original: Pointwise label uncertainty is calculated as
Revised: Pointwise label uncertainty is calculated as

Original: [Equation]
Revised: [Equation]

Original: otherwise (
Revised: otherwise (

Original: [Equation]
Revised: [Equation]

Original: So f ( (cid:15) ) in Eq.
Revised: So f ( (cid:15) ) in Eq.

Original: 7 becomes f ( (cid:15) )
Revised: 7 becomes f ( (cid:15) )

Original: =
Revised: =

Original: log
Revised: log

Original: [ ( C
Revised: [ ( C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:15)
Revised: ) (cid:15)

Original: [Equation]
Revised: [Equation]

Original: z a ) ] .
Revised: z a ) ] .

Original: That is, samples contribute relatively more to the gradient backpropagation when the estimated (cid:15) is small which indicates there is a high probability that the given label is correct.
Revised: That is, samples contribute relatively more to the gradient backpropagation when the estimated (cid:15) is small which indicates there is a high probability that the given label is correct.

Original: By focusing on these reliable targets during training, we can train a more robust classiﬁer.
Revised: By focusing on these reliable targets during training, we can train a more robust classiﬁer.

Original: From this, we can see that the key is to approximate (cid:15) ( z a ) precisely.
Revised: From this, we can see that the key is to approximate (cid:15) ( z a ) precisely.

Original: In the following subsection, we discuss the proposed approach to model (cid:15) ( z a ) using the differences in predictions from the updating classiﬁer model and its moving average model.
Revised: In the following subsection, we discuss the proposed approach to model (cid:15) ( z a ) using the differences in predictions from the updating classiﬁer model and its moving average model.

Original: We want to emphasize that although UMN is based on VAE framework, its architecture can be ﬂexible.
Revised: We want to emphasize that although UMN is based on VAE framework, its architecture can be ﬂexible.

Original: Depending on the task, the encoder of its generative model-VAE can be implemented using various deep learning networks, e.g., AlexNet, GoogLeNet, ResNet, etc.
Revised: Depending on the task, the encoder of its generative model-VAE can be implemented using various deep learning networks, e.g., AlexNet, GoogLeNet, ResNet, etc.

Original: Meanwhile, UMN is not limited to one speciﬁc task, like image classiﬁcation.
Revised: Meanwhile, UMN is not limited to one speciﬁc task, like image classiﬁcation.

Original: It can be easily extended to other classiﬁcation problems.
Revised: It can be easily extended to other classiﬁcation problems.

Original: 3.2. U NCERTAINTY E STIMATION
Revised: 3.2. U NCERTAINTY E STIMATION

Original: As discussed above, we aim to train a more robust classiﬁer in an end-to-end semi-supervised architecture.
Revised: As discussed above, we aim to train a more robust classiﬁer in an end-to-end semi-supervised architecture.

Original: The key to this idea is to estimate sample-wise label noise ( (cid:15) ( z a ) ) as this cannot be known as a priori.
Revised: The key to this idea is to estimate sample-wise label noise ( (cid:15) ( z a ) ) as this cannot be known as a priori.

Original: We are motivated by the initial work of the iterate average model (Polyak &
Revised: We are motivated by the initial work of the iterate average model (Polyak &

Original: Juditsky, 1992) where exponential moving average of the model weights can be used to stabilize the training in stochastic gradient based approaches.
Revised: Juditsky, 1992) where exponential moving average of the model weights can be used to stabilize the training in stochastic gradient based approaches.

Original: Exponential moving average of the model weights can be calculated as:
Revised: Exponential moving average of the model weights can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: (10) where θ is the set of weight parameters of the classiﬁer model and θ (cid:48) is its exponential moving average.
Revised: (10) where θ is the set of weight parameters of the classiﬁer model and θ (cid:48) is its exponential moving average.

Original: γ controls the smoothness of model updates.
Revised: γ controls the smoothness of model updates.

Original: We name them Learner and Guider models respectively.
Revised: We name them Learner and Guider models respectively.

Original: t is the step index of the iterative optimization.
Revised: t is the step index of the iterative optimization.

Original: Since this iterate average gives optimal bound for convergence rate (Polyak & Juditsky, 1992) and can be less sensitive to the noisy updates (Eq. 10), we adopt it as the Guider model to estimate the label uncertainty.
Revised: Since this iterate average gives optimal bound for convergence rate (Polyak & Juditsky, 1992) and can be less sensitive to the noisy updates (Eq. 10), we adopt it as the Guider model to estimate the label uncertainty.

Original: We approximate the label uncertainty via the absolute difference in the predicted probability of the
Revised: We approximate the label uncertainty via the absolute difference in the predicted probability of the

Original: Learner and the Guider for the observed class as following
Revised: Learner and the Guider for the observed class as following

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48) and f indicate the classiﬁer’s predictions of the learner and guider’s respectively.
Revised: where f (cid:48) and f indicate the classiﬁer’s predictions of the learner and guider’s respectively.

Original: As the
Revised: As the

Original: Learner learns from the noisy labeled data, the Guider model is used to weigh the samples based on uncertainty thus limiting the contributions of unreliable samples to the gradients back propagated.
Revised: Learner learns from the noisy labeled data, the Guider model is used to weigh the samples based on uncertainty thus limiting the contributions of unreliable samples to the gradients back propagated.

Original: In the following subsection, we provide the analysis of the reason why using the difference between the predictions of the Guider and Learner models is a good measure for the uncertainty estimation.
Revised: In the following subsection, we provide the analysis of the reason why using the difference between the predictions of the Guider and Learner models is a good measure for the uncertainty estimation.

Original: 3.2. T HEORETICAL A NALYSIS
Revised: 3.2. T HEORETICAL A NALYSIS

Original: Our deep generative model allows us to model the correct labels via the observed labels and therefore leads to a plausible way to adjust sample weights during training.
Revised: Our deep generative model allows us to model the correct labels via the observed labels and therefore leads to a plausible way to adjust sample weights during training.

Original: We will show in the following theorem that the label correctness is associated with the difference between the Learner and Guider models during training.
Revised: We will show in the following theorem that the label correctness is associated with the difference between the Learner and Guider models during training.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that θ ∗ is the optimal solution of the optimization problem.
Revised: Suppose that θ ∗ is the optimal solution of the optimization problem.

Original: θ L is the stochastic random variable for SGD (learner) and θ G is iterate average (guider) of θ L , θ G ( T ) =
Revised: θ L is the stochastic random variable for SGD (learner) and θ G is iterate average (guider) of θ L , θ G ( T ) =

Original: µ T ≡ 1 T (cid:82) T 0 θ L ( t ) dt .
Revised: µ T ≡ 1 T (cid:82) T 0 θ L ( t ) dt .

Original: The distance between θ L and θ ∗ is consistent with the distance between θ L and θ G in the sense that
Revised: The distance between θ L and θ ∗ is consistent with the distance between θ L and θ G in the sense that

Original: [Equation]
Revised: [Equation]

Original: which holds when the training epoch T satisﬁes the following relation
Revised: which holds when the training epoch T satisﬁes the following relation

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where λ is the constant learning rate.
Revised: where λ is the constant learning rate.

Original: We can see from Eq.12, during the early training stage, the distance between the learner and the optimal solution increases when the distance between the learner and guider increases and vice versa.
Revised: We can see from Eq.12, during the early training stage, the distance between the learner and the optimal solution increases when the distance between the learner and guider increases and vice versa.

Original: This leads to preventing samples with uncertain labels to participate in training and thus reducing the overﬁtting on potentially wrong labels.
Revised: This leads to preventing samples with uncertain labels to participate in training and thus reducing the overﬁtting on potentially wrong labels.

Original: The proof of the theorem is given in the Appendix.
Revised: The proof of the theorem is given in the Appendix.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Our experiments are designed to evaluate whether UMN is an effective approach to learn a good model with limited annotated training data which include label noise.
Revised: Our experiments are designed to evaluate whether UMN is an effective approach to learn a good model with limited annotated training data which include label noise.

Original: We compare to popular approaches of supervised learning, semi-supervised learning and robust learning dealing with noisy labels.
Revised: We compare to popular approaches of supervised learning, semi-supervised learning and robust learning dealing with noisy labels.

Original: These methods represent state-of-the-art approaches for model learning with noisy data.
Revised: These methods represent state-of-the-art approaches for model learning with noisy data.

Original: Moreover, we also compare the results with the pipeline without using our uncertainty estimation module.
Revised: Moreover, we also compare the results with the pipeline without using our uncertainty estimation module.

Original: Further, we also evaluate the performance of applying the uncertainty estimation module in identifying corrupted labels.
Revised: Further, we also evaluate the performance of applying the uncertainty estimation module in identifying corrupted labels.

Original: We experiment on a variety of image classiﬁcation problems with varying degrees of label corruption rates.
Revised: We experiment on a variety of image classiﬁcation problems with varying degrees of label corruption rates.

Original: We use three popular datasets including MNIST , SVHN and CIFAR-10 .
Revised: We use three popular datasets including MNIST , SVHN and CIFAR-10 .

Original: For a comprehensive evaluation, we set up ﬁve different uniform labels corruption rates including [10% , 20% , 30% , 40% , 50%] .
Revised: For a comprehensive evaluation, we set up ﬁve different uniform labels corruption rates including [10% , 20% , 30% , 40% , 50%] .

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: In the experiments, supervised deep learning framework means all the label data are directly used for training.
Revised: In the experiments, supervised deep learning framework means all the label data are directly used for training.

Original: For SVHN and CIFAR-10, we experiment a CNN architecture with 13 convolutional neural network as well as the ResNet-101 architecture (He et al., 2015) as baselines for supervised learning approach.
Revised: For SVHN and CIFAR-10, we experiment a CNN architecture with 13 convolutional neural network as well as the ResNet-101 architecture (He et al., 2015) as baselines for supervised learning approach.

Original: To compare with semi-supervised learning approaches, we select two recent popular works.
Revised: To compare with semi-supervised learning approaches, we select two recent popular works.

Original: One is named as mean-teacher (MT) proposed by Tarvainen and Valpola (Tarvainen & Valpola, 2017) and the other one is Langevin et al.
Revised: One is named as mean-teacher (MT) proposed by Tarvainen and Valpola (Tarvainen & Valpola, 2017) and the other one is Langevin et al.

Original: (Langevin et al., 2018) named Mislabeled-VAE (M-VAE).
Revised: (Langevin et al., 2018) named Mislabeled-VAE (M-VAE).

Original: Langevin et al.
Revised: Langevin et al.

Original: (Langevin et al., 2018) only roughly discusses their idea without showing the details of the model architecture and experimental results on popular benchmarks.
Revised: (Langevin et al., 2018) only roughly discusses their idea without showing the details of the model architecture and experimental results on popular benchmarks.

Original: In this work, we implement the idea in (Langevin et al., 2018) and compare with UMN in all three different datasets.
Revised: In this work, we implement the idea in (Langevin et al., 2018) and compare with UMN in all three different datasets.

Original: In our experiments, we use the same encoder and decoder architectures for UMN and M-VAE.
Revised: In our experiments, we use the same encoder and decoder architectures for UMN and M-VAE.

Original: In comparison to robust learning approaches, we apply another two recent works including MentorNet (Jiang et al., 2017) and Reweight (Ren et al., 2018).
Revised: In comparison to robust learning approaches, we apply another two recent works including MentorNet (Jiang et al., 2017) and Reweight (Ren et al., 2018).

Original: For a fair comparison with these works, we follow all the original setting in these papers in our experiments.
Revised: For a fair comparison with these works, we follow all the original setting in these papers in our experiments.

Original: For each corruption ratio, we ranexperiments by randomly choosing samples to corrupt each time, and report the mean error rate.
Revised: For each corruption ratio, we ranexperiments by randomly choosing samples to corrupt each time, and report the mean error rate.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1, our framework (UMN) is composed of two encoders and two decoders (VAE and C-VAE) along with a classiﬁer.
Revised: 1, our framework (UMN) is composed of two encoders and two decoders (VAE and C-VAE) along with a classiﬁer.

Original: For the experiments in MNIST, We adapt a multi-layer perceptron (MLP) architecture similar to the one proposed in (Kingma et al., 2014), for the VAE, C-VAE and the classiﬁer models.
Revised: For the experiments in MNIST, We adapt a multi-layer perceptron (MLP) architecture similar to the one proposed in (Kingma et al., 2014), for the VAE, C-VAE and the classiﬁer models.

Original: The encoders, decoders and the classiﬁer of our model use a single hidden layer with ReLU activation and an output layer with no activation.
Revised: The encoders, decoders and the classiﬁer of our model use a single hidden layer with ReLU activation and an output layer with no activation.

Original: For SVHN and CIFAR-10, we apply a 13-layer convolutional neural network (ConvNet) to build our framework.
Revised: For SVHN and CIFAR-10, we apply a 13-layer convolutional neural network (ConvNet) to build our framework.

Original: At the ﬁrst stage of unsupervised learning, z a is generated via convolution and up-sampling layers.
Revised: At the ﬁrst stage of unsupervised learning, z a is generated via convolution and up-sampling layers.

Original: Similar as the ﬁrst stage, z b is generated via the encoder framework which is composed of convolution and dense layers.
Revised: Similar as the ﬁrst stage, z b is generated via the encoder framework which is composed of convolution and dense layers.

Original: We maintain a moving average of the learner and use the difference between the predictions of learner and guider model to calculate (cid:15) .
Revised: We maintain a moving average of the learner and use the difference between the predictions of learner and guider model to calculate (cid:15) .

Original: The decoder in the semi-supervised learning stage is built with de-convolution and dense layers.
Revised: The decoder in the semi-supervised learning stage is built with de-convolution and dense layers.

Original: Due to the limitation in space, we detail the model architecture in the Appendix section.
Revised: Due to the limitation in space, we detail the model architecture in the Appendix section.

Original: We use the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.001, β 1 = 0 .
Revised: We use the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.001, β 1 = 0 .

Original: 90 and β 2 = 0 .
Revised: 90 and β 2 = 0 .

Original: Each experiment has 5 runs and each run has 350 epochs.
Revised: Each experiment has 5 runs and each run has 350 epochs.

Original: In each epoch, UMN uses all the training data including labeled and unlabeled data.
Revised: In each epoch, UMN uses all the training data including labeled and unlabeled data.

Original: For these supervised learning frameworks, we only use the labeled data.
Revised: For these supervised learning frameworks, we only use the labeled data.

Original: Our implementation is based on deep learning framework TensorFlow with a single NVIDIA Tesla P100 GPU.
Revised: Our implementation is based on deep learning framework TensorFlow with a single NVIDIA Tesla P100 GPU.

Original: 4. D ATASETS AND R ESULTS 4.2. MNIST
Revised: 4. D ATASETS AND R ESULTS 4.2. MNIST

Original: MNIST is a widely used dataset in machine learning community.
Revised: MNIST is a widely used dataset in machine learning community.

Original: It includes 60 , 000 images with size 28 × 28 pixels.
Revised: It includes 60 , 000 images with size 28 × 28 pixels.

Original: In our experiments, we use 50 , 000 images for training and 10 , 000 images for testing.
Revised: In our experiments, we use 50 , 000 images for training and 10 , 000 images for testing.

Original: 100 labeled data samples are used in the experiments.
Revised: 100 labeled data samples are used in the experiments.

Original: In semi-supervised training, each mini-batch have 5 labeled samples and 95 unlabeled examples.
Revised: In semi-supervised training, each mini-batch have 5 labeled samples and 95 unlabeled examples.

Original: In our compared supervised model, we apply a
Revised: In our compared supervised model, we apply a

Original: Multi-Layer Perceptron (MLP) classiﬁer with one hidden layer of 784 units with ReLU activation.
Revised: Multi-Layer Perceptron (MLP) classiﬁer with one hidden layer of 784 units with ReLU activation.

Original: All the results are listed in Table 1.
Revised: All the results are listed in Table 1.

Original: From the results, we can ﬁnd that UMN performs much better than other approaches.
Revised: From the results, we can ﬁnd that UMN performs much better than other approaches.

Original: Besides UMN,
Revised: Besides UMN,

Original: M-VAE (Langevin et al., 2018) outperforms other benchmarks.
Revised: M-VAE (Langevin et al., 2018) outperforms other benchmarks.

Original: One possible reason may be due to probablistic modeling of the uncertainty.
Revised: One possible reason may be due to probablistic modeling of the uncertainty.

Original: However, M-VAE depends on the pre-deﬁned label corruption ratio which can’t be obtained in most real scenarios.
Revised: However, M-VAE depends on the pre-deﬁned label corruption ratio which can’t be obtained in most real scenarios.

Original: We did not run Reweight (Ren et al., 2018) on the MNIST dataset since at the time of running the experiments an implementation of the approach on noisy MNIST data was not available and our implementation of the approach would not be a fair comparison without having the right parameter settings.
Revised: We did not run Reweight (Ren et al., 2018) on the MNIST dataset since at the time of running the experiments an implementation of the approach on noisy MNIST data was not available and our implementation of the approach would not be a fair comparison without having the right parameter settings.

Original: However, comparisons are shown on the other datasets as implementation for them was made available by the authors.
Revised: However, comparisons are shown on the other datasets as implementation for them was made available by the authors.

Original: 4.2. SVHN
Revised: 4.2. SVHN

Original: We experiment with the Street View House Numbers (SVHN) datasets.
Revised: We experiment with the Street View House Numbers (SVHN) datasets.

Original: This dataset includes 73 ,
Revised: This dataset includes 73 ,

Original: RGB images of 32 × 32 resolution belonging to ten different classes.
Revised: RGB images of 32 × 32 resolution belonging to ten different classes.

Original: Following the same experimental settings as in (Tarvainen & Valpola, 2017), we use 500 labeled data samples from SVHN wheredata samples per category and we use the rest of the images for unlabeled data in the semi-supervised learning setting.
Revised: Following the same experimental settings as in (Tarvainen & Valpola, 2017), we use 500 labeled data samples from SVHN wheredata samples per category and we use the rest of the images for unlabeled data in the semi-supervised learning setting.

Original: We randomly corrupt the sample labels within each category uniformly with our deﬁned corruption ratio.
Revised: We randomly corrupt the sample labels within each category uniformly with our deﬁned corruption ratio.

Original: We also compare with other ﬁve different models on SVHN dataset.
Revised: We also compare with other ﬁve different models on SVHN dataset.

Original: In our experiment, each batch includes 5% labeled data.
Revised: In our experiment, each batch includes 5% labeled data.

Original: The comparison results are given in Table 1.
Revised: The comparison results are given in Table 1.

Original: As listed in this table, we can ﬁnd that UMN behaves the best in general except for the case when the corruption ratio is 10% .
Revised: As listed in this table, we can ﬁnd that UMN behaves the best in general except for the case when the corruption ratio is 10% .

Original: However, we do not observe this phenomena in MNIST and CIFAR-10.
Revised: However, we do not observe this phenomena in MNIST and CIFAR-10.

Original: One of the reasons may be that MT can handle the situation when the noisy data occupy a small portion.
Revised: One of the reasons may be that MT can handle the situation when the noisy data occupy a small portion.

Original: The performance of MT model drops signiﬁcantly as we ingest more mislabeled data.
Revised: The performance of MT model drops signiﬁcantly as we ingest more mislabeled data.

Original: From the result, we can also observe that other two robust learning
Revised: From the result, we can also observe that other two robust learning

Original: [Table]
Revised: [Table]

Original: approaches (MentorNet and Reweight) achieve better performance than supervised only but much lower than these Semi-supervised learning methods.
Revised: approaches (MentorNet and Reweight) achieve better performance than supervised only but much lower than these Semi-supervised learning methods.

Original: One possible reason is that they do not utilize the unlabeled data.
Revised: One possible reason is that they do not utilize the unlabeled data.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: [Table]
Revised: [Table]

Original: overﬁtting is observed in the M-VAE at a higher ratio of corruption.
Revised: overﬁtting is observed in the M-VAE at a higher ratio of corruption.

Original: As we discussed previously
Revised: As we discussed previously

Original: [Table]
Revised: [Table]

Original: one major reason is because of the pre-deﬁned f ( (cid:15) ) which is used as the constant value during the training phase.
Revised: one major reason is because of the pre-deﬁned f ( (cid:15) ) which is used as the constant value during the training phase.

Original: In contrast, UMN applies the sample-wise uncertainty estimation.
Revised: In contrast, UMN applies the sample-wise uncertainty estimation.

Original: It is accurate and more close to the real scenario.
Revised: It is accurate and more close to the real scenario.

Original: Furthermore, the training of UMN also converges faster with better performance.
Revised: Furthermore, the training of UMN also converges faster with better performance.

Original: Moreover
Revised: Moreover

Original: [Table]
Revised: [Table]

Original: we conduct the experiments and evaluate the model performances using varying numbers of labeled data with the SVHN.
Revised: we conduct the experiments and evaluate the model performances using varying numbers of labeled data with the SVHN.

Original: We compare the performance for a popular supervised robust learning approach MentorNet(MN) and UMN, for which the test errors shown in Table 2.
Revised: We compare the performance for a popular supervised robust learning approach MentorNet(MN) and UMN, for which the test errors shown in Table 2.

Original: In UMN
Revised: In UMN

Original: [Table]
Revised: [Table]

Original: we still utilize all the unlabeled data which are not utilized for labeled data.
Revised: we still utilize all the unlabeled data which are not utilized for labeled data.

Original: UMN shows better performance for labeled data less than 1000 labels per class, which suggests the important role of the semi-supervised learning in robust learning and hence justify the purpose of this work.
Revised: UMN shows better performance for labeled data less than 1000 labels per class, which suggests the important role of the semi-supervised learning in robust learning and hence justify the purpose of this work.

Original: [Table]
Revised: [Table]

Original: Next, we run the comparisions on CIFAR-10 dataset (Krizhevsky, 2009).
Revised: Next, we run the comparisions on CIFAR-10 dataset (Krizhevsky, 2009).

Original: This dataset contains 32 × 32 pixels RGB images belonging to ten different classes.
Revised: This dataset contains 32 × 32 pixels RGB images belonging to ten different classes.

Original: To have a fair comparison, we follow the same experimental setting as used in (Tarvainen & Valpola, 2017).
Revised: To have a fair comparison, we follow the same experimental setting as used in (Tarvainen & Valpola, 2017).

Original: We randomly select 100 data samples from each category.
Revised: We randomly select 100 data samples from each category.

Original: In total, 1 , 000 labeled data samples are included in the training set.
Revised: In total, 4 , 000 labeled data samples are included in the training set.

Original: The rest of the training data are used for unlabeled data.
Revised: The rest of the training data are used for unlabeled data.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: From the listed results, we can see that UMN achieves the best performance among all experiments.
Revised: From the listed results, we can see that UMN achieves the best performance among all experiments.

Original: [Figure]
Revised: [Figure]

Original: 4. I DENTIFYING SAMPLES WITH CORRUPT LABELS
Revised: 4. I DENTIFYING SAMPLES WITH CORRUPT LABELS

Original: We have conducted experiments using the MNIST-digit dataset to quantify how well UMN can identify samples whose labels are corrupted.
Revised: We have conducted experiments using the MNIST-digit dataset to quantify how well UMN can identify samples whose labels are corrupted.

Original: We say, a sample’s label is corrupted if there is a disagreement of the predictions between the Learner and the Guider models.
Revised: We say, a sample’s label is corrupted if there is a disagreement of the predictions between the Learner and the Guider models.

Original: That is, if the estimated (cid:15) > 

Revised: That is, if the estimated (cid:15) >

Original: (Eq.
Revised: (Eq.

Original: 11) for a given sample.
Revised: 11) for a given sample.

Original: Table 3 summarizes the accuracy in identifying samples with corrupt labels.
Revised: Table 3 summarizes the accuracy in identifying samples with corrupt labels.

Original: The sample-wise estimate of (cid:15) estimated in the ﬁnal epoch of training is used in the analysis.
Revised: The sample-wise estimate of (cid:15) estimated in the ﬁnal epoch of training is used in the analysis.

Original: From the listed results, we can see that UMN has demonstrated very promising potential for ﬁltering out noisy data.
Revised: From the listed results, we can see that UMN has demonstrated very promising potential for ﬁltering out noisy data.

Original: It could be easily adopted to other related works as the data pre-processing step.
Revised: It could be easily adopted to other related works as the data pre-processing step.

Original: Fig.
Revised: Fig.

Original: 4b shows the progression of the uncertainty estimate (cid:15) while training on CIFAR-10 dataset with a label corruption rate of 20% .
Revised: 4b shows the progression of the uncertainty estimate (cid:15) while training on CIFAR-10 dataset with a label corruption rate of 20% .

Original: We can see that the distribution is bi-modal and that the bi-modal separation increases as the training progresses.
Revised: We can see that the distribution is bi-modal and that the bi-modal separation increases as the training progresses.

Original: i.e., our approach becomes more conﬁdent in differentiating samples with corrupt labels from samples with correct labels.
Revised: i.e., our approach becomes more conﬁdent in differentiating samples with corrupt labels from samples with correct labels.

Original: In addition, to better understand the improved performance of UMN in these experiments.
Revised: In addition, to better understand the improved performance of UMN in these experiments.

Original: We compare the structure of high-dimensional latent feature representations obtained from two VAE components between the model with and the one without using label uncertainty estimation module.
Revised: We compare the structure of high-dimensional latent feature representations obtained from two VAE components between the model with and the one without using label uncertainty estimation module.

Original: We apply t-SNE to the latent feature vector ( z a ) on CIFAR10 test dataset.
Revised: We apply t-SNE to the latent feature vector ( z a ) on CIFAR10 test dataset.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4a, it is clear that the t-SNE maps exhibit different distributions between two architectures.
Revised: 4a, it is clear that the t-SNE maps exhibit different distributions between two architectures.

Original: In particular, as indicated by the feature distribution within the black and red boxes,
Revised: In particular, as indicated by the feature distribution within the black and red boxes,

Original: UMN produces a much better separate map between different classes.
Revised: UMN produces a much better separate map between different classes.

Original: The used training data had a label corruption rate of 30% .
Revised: The used training data had a label corruption rate of 30% .

Original: In Fig.
Revised: In Fig.

Original: 5 we show the distribution of the embeddings for the training samples using different approaches.
Revised: 5 we show the distribution of the embeddings for the training samples using different approaches.

Original: t-SNE is used for visualizing in 2-dimensions.
Revised: t-SNE is used for visualizing in 2-dimensions.

Original: The training data has 100 labeled samples with 50% label corruption.
Revised: The training data has 100 labeled samples with 50% label corruption.

Original: We can see that the supervised approach clearly overﬁts on the noisy labels.
Revised: We can see that the supervised approach clearly overﬁts on the noisy labels.

Original: The semi-supervised approach used is better but one can still see that some of the classes are confused with each other due to label noise, especially when looking at the central region of the plot.
Revised: The semi-supervised approach used is better but one can still see that some of the classes are confused with each other due to label noise, especially when looking at the central region of the plot.

Original: However, it can be seen that using the proposed UMN approach results in better intra-class grouping and inter-class separation of training data
Revised: However, it can be seen that using the proposed UMN approach results in better intra-class grouping and inter-class separation of training data

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: n 0.
Revised: n 0.

Original: E p o c h
Revised: E p o c h

Original: N u m b e r o f l a b e l e d d a t a
Revised: N u m b e r o f l a b e l e d d a t a

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we target a new problem of learning on partially labeled data with noise and propose a novel framework (UMN).
Revised: In this work we target a new problem of learning on partially labeled data with noise and propose a novel framework (UMN).

Original: UMN is a semi-supervised deep generative model dealing with data including scarce and potentially corrupted labels.
Revised: UMN is a semi-supervised deep generative model dealing with data including scarce and potentially corrupted labels.

Original: Moreover, UMN can explicitly estimate the label uncertainty for a given potentially mislabeled data sample.
Revised: Moreover, UMN can explicitly estimate the label uncertainty for a given potentially mislabeled data sample.

Original: Furthermore, we integrate these frameworks into an end-to-end pipeline.
Revised: Furthermore, we integrate these frameworks into an end-to-end pipeline.

Original: Compared to previous works, we do not need the subset of clean label data or any preknowledge of the corruption ratio.
Revised: Compared to previous works, we do not need the subset of clean label data or any preknowledge of the corruption ratio.

Original: UMN is able to estimate the label correctness based on the uncertainty calculation.
Revised: UMN is able to estimate the label correctness based on the uncertainty calculation.

Original: Experimental results demonstrate the superiority of UMN over state-of-the-arts for training deep learning models with noisy labels.
Revised: Experimental results demonstrate the superiority of UMN over state-of-the-arts for training deep learning models with noisy labels.

Original:  A PPENDIX 6. ELBO FOR VAE OF U NCERTAINTY M ININING N ET ( UMN )
Revised:  A PPENDIX 6. ELBO FOR VAE OF U NCERTAINTY M ININING N ET ( UMN )

Original: We start from the Jensen’s inequality for the log probability of the observations,
Revised: We start from the Jensen’s inequality for the log probability of the observations,

Original: [Equation]
Revised: [Equation]

Original: (14) where the observed variables are X
Revised: (14) where the observed variables are X

Original: = { ˆ y, X } and the latent variables v for the VAE structure we proposed with a stacked semi-supervised learning architecture.
Revised: = { ˆ y, X } and the latent variables v for the VAE structure we proposed with a stacked semi-supervised learning architecture.

Original: The posterior and likelihood factorizes as
Revised: The posterior and likelihood factorizes as

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss on r.h.s of Eq.
Revised: The reconstruction loss on r.h.s of Eq.

Original: 14 then gives,
Revised: 14 then gives,

Original: [Equation]
Revised: [Equation]

Original: where q φ and p θ are the encoder and decoder respectively for the variational autoencoder and can be parametrized by deep neural network.
Revised: where q φ and p θ are the encoder and decoder respectively for the variational autoencoder and can be parametrized by deep neural network.

Original: We note that the second term in the last equality of Eq.
Revised: We note that the second term in the last equality of Eq.

Original: 15 is summed over the labeled samples in Semi-supervised learning.
Revised: 15 is summed over the labeled samples in Semi-supervised learning.

Original: The KL divergence in Eq.
Revised: The KL divergence in Eq.

Original: 14 is given
Revised: 14 is given

Original: [Equation]
Revised: [Equation]

Original: To summarize
Revised: To summarize

Original: [Equation]
Revised: [Equation]

Original: the ELBO for VAE of UMN is
Revised: the ELBO for VAE of UMN is

Original: [Equation]
Revised: [Equation]

Original: E q φ ( z a | x ) (cid:88)
Revised: E q φ ( z a | x ) (cid:88)

Original: y ∈
Revised: y ∈

Original: C q φ ( y | z a ) E q φ ( z b |
Revised: C q φ ( y | z a ) E q φ ( z b |

Original: z
Revised: z

Original: a ,y ) (cid:0) log p θ ( z a | z b , y ) − log q φ ( z a | x ) (cid:1) + E q φ
Revised: a ,y ) (cid:0) log p θ ( z a | z b , y ) − log q φ ( z a | x ) (cid:1) + E q φ

Original: ( z a | x ) log P ( x | z a ) + E q φ ( z a | x ) (cid:88)
Revised: ( z a | x ) log P ( x | z a ) + E q φ ( z a | x ) (cid:88)

Original: y k ∈ C q φ ( y k | z a ) log P θ (ˆ y | y k , z a ) , (18) where in the last term, estimations for probabilities of the observed label is given based on the true labels and the corresponding encoded sample z a .
Revised: y k ∈ C q φ ( y k | z a ) log P θ (ˆ y | y k , z a ) , (18) where in the last term, estimations for probabilities of the observed label is given based on the true labels and the corresponding encoded sample z a .

Original: The loss given by the ELBO are summed over all the samples except for the last term is summed over the samples with the observed variable ˆ y available.
Revised: The loss given by the ELBO are summed over all the samples except for the last term is summed over the samples with the observed variable ˆ y available.

Original: 6. A LGORITHM OF UMN
Revised: 6. A LGORITHM OF UMN

Original: while Traning() do
Revised: while Traning() do

Original: Sample a minibatch from dataset S i ;
Revised: Sample a minibatch from dataset S i ;

Original: Draw
Revised: Draw

Original: z ia from q φ ( z ia | x i ) given x i ∈ S i ;
Revised: z ia from q φ ( z ia | x i ) given x i ∈ S i ;

Original: Draw
Revised: Draw

Original: z ia from q φ ( z ia | x i ) ; for all data x i ∈ S i : do y i ∼ q bφ ( y i | z ia ) end for all labeled data x iL ∈ S i : do ˆ
Revised: z ia from q φ ( z ia | x i ) ; for all data x i ∈ S i : do y i ∼ q bφ ( y i | z ia ) end for all labeled data x iL ∈ S i : do ˆ

Original: y i ∼ p θ (ˆ y i | y, z ia ) Compute (cid:15) based on the uncertainty estimation Eq.(9) via Learner and
Revised: y i ∼ p θ (ˆ y i | y, z ia ) Compute (cid:15) based on the uncertainty estimation Eq.(9) via Learner and

Original: Guider end
Revised: Guider end

Original: Compute the generative process via Eq.(3) ;
Revised: Compute the generative process via Eq.(3) ;

Original: Compute loss function L via Eq.(5) ; g θ =
Revised: Compute loss function L via Eq.(5) ; g θ =

Original: ∂
Revised: ∂

Original: L ∂θ ; g φ = ∂
Revised: L ∂θ ; g φ = ∂

Original: L ∂φ ; θ = θ - AdamUpdate( θ ); φ = φ - AdamUpdate( φ );
Revised: L ∂φ ; θ = θ - AdamUpdate( θ ); φ = φ - AdamUpdate( φ );

Original: For φ in classiﬁer y i ∼ q bφ
Revised: For φ in classiﬁer y i ∼ q bφ

Original: ( y i | x i ) , maintain Polyak’s moving average for the guider as φ G ( t ) =
Revised: ( y i | x i ) , maintain Polyak’s moving average for the guider as φ G ( t ) =

Original: (1 − α )
Revised: (1 − α )

Original: φ L ( t − 1)
Revised: φ L ( t − 1)

Original: + α ∗ φ L ( t ) end
Revised: + α ∗ φ L ( t ) end

Original: 6. M ODEL A RCHITECTURE OF UMN WITH C ONVOLUTIONAL N EURAL N ETWORK
Revised: 6. M ODEL A RCHITECTURE OF UMN WITH C ONVOLUTIONAL N EURAL N ETWORK

Original: The model architecture is listed in the table 4.
Revised: The model architecture is listed in the table 4.

Original: The plot describe how the classiﬁer and VAE model is setup.
Revised: The plot describe how the classiﬁer and VAE model is setup.

Original: The VAE is a stacked autoencoder with M1 + M2 Kingma et al.
Revised: The VAE is a stacked autoencoder with M1 + M2 Kingma et al.

Original: M1 is a variational autoencoder with convolution neural networks as the encoder and decoder.
Revised: M1 is a variational autoencoder with convolution neural networks as the encoder and decoder.

Original: M2 is composed of only fully connected layers.
Revised: M2 is composed of only fully connected layers.

Original: [Table]
Revised: [Table]

Original: Fully
Revised: Fully

Original: Connected
Revised: Connected

Original: [Table]
Revised: [Table]

Original: Softmax 128 →
Revised: Softmax 128 →

Original: [Table]
Revised: [Table]

Original: Hyperparameters
Revised: Hyperparameters

Original: Conv2D ﬁlter size: (3, 3, 32), same padding
Revised: Conv2D ﬁlter size: (3, 3, 32), same padding

Original: Flatten (
Revised: Flatten (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 32) →
Revised: 32) →

Original: Fully
Revised: Fully

Original: Connected
Revised: Connected

Original: 1152 →
Revised: 1152 →

Original: Fully
Revised: Fully

Original: Connected
Revised: Connected

Original: 256 →
Revised: 256 →

Original: Reshape 1152 → (6 ,
Revised: Reshape 1152 → (6 ,

Original: [Table]
Revised: [Table]

Original: Conv2D
Revised: Conv2D

Original: Transpose
Revised: Transpose

Original: (6 , 6 , 32) → (8 , 8 , 128)
Revised: (6 , 6 , 32) → (8 , 8 , 128)

Original: Up
Revised: Up

Original: sampling
Revised: sampling

Original: Conv2D ﬁlter size: (3, 3, 128), same padding
Revised: Conv2D ﬁlter size: (3, 3, 128), same padding

Original: Conv2D ﬁlter size: (3, 3, 128), same padding Conv2D ﬁlter size: (3, 3, 128), same padding
Revised: Conv2D ﬁlter size: (3, 3, 128), same padding Conv2D ﬁlter size: (3, 3, 128), same padding

Original: Up-sampling
Revised: Up-sampling

Original: Conv2D (shared
Revised: Conv2D (shared

Original: with
Revised: with

Original: MVAE ﬁlter size: (3, 3, 128), same padding
Revised: MVAE ﬁlter size: (3, 3, 128), same padding

Original: Conv2D (shared with MVAE)
Revised: Conv2D (shared with MVAE)

Original: ﬁlter size: (3, 3, 128), same padding
Revised: ﬁlter size: (3, 3, 128), same padding

Original: Conv2D (shared with MVAE)
Revised: Conv2D (shared with MVAE)

Original: ﬁlter size: (3, 3, 128), same padding
Revised: ﬁlter size: (3, 3, 128), same padding

Original: 6. U NCERTAINTY E STIMATION AS D IFFERENCE BETWEEN THE G UIDER AND THE L EARNER
Revised: 6. U NCERTAINTY E STIMATION AS D IFFERENCE BETWEEN THE G UIDER AND THE L EARNER

Original: We illustrate this using with a model near the convergence in a convex domain (Mandt et al., 2017), assuming a small label corruption ratio.
Revised: We illustrate this using with a model near the convergence in a convex domain (Mandt et al., 2017), assuming a small label corruption ratio.

Original: This suggests that our calculated uncertainty can help focus more on the correct labeled samples.
Revised: This suggests that our calculated uncertainty can help focus more on the correct labeled samples.

Original: Assumption 1 We assume that in the optimization problem, the stationary distribution is constrained to a convex region, where the loss function has a quadratic form
Revised: Assumption 1 We assume that in the optimization problem, the stationary distribution is constrained to a convex region, where the loss function has a quadratic form

Original: [Equation]
Revised: [Equation]

Original: (19) where H is the Hessian of the loss surface near minimum and the vector θ of the model parameters.
Revised: (19) where H is the Hessian of the loss surface near minimum and the vector θ of the model parameters.

Original: Without loss of generality, optimal θ lies on the origin.
Revised: Without loss of generality, optimal θ lies on the origin.

Original: Input (32, 32, 3)
Revised: Input (32, 32, 3)

Original: Pool 2d (2, 2)
Revised: Pool 2d (2, 2)

Original: Conv2D (64, 3,3) X
Revised: Conv2D (64, 3,3) X

Original: Conv2D (128, 3,3) X
Revised: Conv2D (128, 3,3) X

Original: Pool 2d (2, 2)
Revised: Pool 2d (2, 2)

Original: Conv 2D X 3 (3,3,128) (Z1)
Revised: Conv 2D X 3 (3,3,128) (Z1)

Original: C o n v 2 D
Revised: C o n v 2 D

Original: X 3 , ( 3 , 3 , 32 )
Revised: X 3 , ( 3 , 3 , 32 )

Original: F l a tt e n (16, 16, 64) (1152)
Revised: F l a tt e n (16, 16, 64) (1152)

Original: D e n s e ( Z 2 )
Revised: D e n s e ( Z 2 )

Original: D e n s e (1152)
Revised: D e n s e (1152)

Original: R e s h a p e (6, 6, 128)
Revised: R e s h a p e (6, 6, 128)

Original: Conv_T2D X 3 (3, 3, 128) (Z1)
Revised: Conv_T2D X 3 (3, 3, 128) (Z1)

Original: Upsampling
Revised: Upsampling

Original: Conv 2D X 3 (128, 3,3)
Revised: Conv 2D X 3 (128, 3,3)

Original: Upsampling
Revised: Upsampling

Original: Conv 2D X 3 (64, 32, 32)
Revised: Conv 2D X 3 (64, 32, 32)

Original: Reconstructed Input (32, 32, 3) (16, 16, 128) (32, 32, 64) (8, 8, 128) (8, 8, 128) (256) (8, 8, 128) (16, 16, 128) (16, 16, 128) (32, 32, 128) (32, 32, 128) (32, 32, 3)
Revised: Reconstructed Input (32, 32, 3) (16, 16, 128) (32, 32, 64) (8, 8, 128) (8, 8, 128) (256) (8, 8, 128) (16, 16, 128) (16, 16, 128) (32, 32, 128) (32, 32, 128) (32, 32, 3)

Original: AVG Pooling (6,6)
Revised: AVG Pooling (6,6)

Original: FCC
Revised: FCC

Original: [Figure]
Revised: [Figure]

Original: MVAE
Revised: MVAE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: X 3 (1, 1,128)
Revised: X 3 (1, 1,128)

Original: [Figure]
Revised: [Figure]

Original: 2D X 3 (1, 1,128) (8, 8, 128)
Revised: 2D X 3 (1, 1,128) (8, 8, 128)

Original: [Figure]
Revised: [Figure]

Original: The network has two modules.
Revised: The network has two modules.

Original: The ﬁrst one is classiﬁer composed of convolution, pooling, dense and softmax layers.
Revised: The ﬁrst one is classiﬁer composed of convolution, pooling, dense and softmax layers.

Original: The second one is an M1 + M2 variational autoencoder.
Revised: The second one is an M1 + M2 variational autoencoder.

Original: The z 1 of M1 + M and the classiﬁer share the same encoder.
Revised: The z 1 of M1 + M and the classiﬁer share the same encoder.

Original: z 1 and z 2 are the two variational autoencoder that encode and decode the raw image.
Revised: z 1 and z 2 are the two variational autoencoder that encode and decode the raw image.

Original: [Figure]
Revised: [Figure]

Original: EHAVIOR AS S TOCHASTIC P ROCESS
Revised: EHAVIOR AS S TOCHASTIC P ROCESS

Original: The convergence behavior of Stochastic Gradient Descent (SGD) can be described by a stochastic differential equation as
Revised: The convergence behavior of Stochastic Gradient Descent (SGD) can be described by a stochastic differential equation as

Original: [Equation]
Revised: [Equation]

Original: 20) where g ( θ ) ≡ H θ ( t ) is the gradient for the weights and S is the mini-batch size.
Revised: 20) where g ( θ ) ≡ H θ ( t ) is the gradient for the weights and S is the mini-batch size.

Original: d W ( t ) represents the Wienner process in the stochastic gradient descent.
Revised: d W ( t ) represents the Wienner process in the stochastic gradient descent.

Original: B ( θ ) is introduced due to the noise in the stochastic gradient descent process.
Revised: B ( θ ) is introduced due to the noise in the stochastic gradient descent process.

Original: The covariance of the SGD process Σ = E ( θθ (cid:62) ) satisﬁes
Revised: The covariance of the SGD process Σ = E ( θθ (cid:62) ) satisﬁes

Original: H Σ + Σ H = λS BB (cid:62) .
Revised: H Σ + Σ H = λS BB (cid:62) .

Original: λ and t represent the step size and step index respectively.
Revised: λ and t represent the step size and step index respectively.

Original: Lemma 2.
Revised: Lemma 2.

Original: The stochastic process for parameter optimization of deep learning problem using only piecewise linear activation function with and without noisy labels can be described by the following
Revised: The stochastic process for parameter optimization of deep learning problem using only piecewise linear activation function with and without noisy labels can be described by the following

Original: Wienner processes,
Revised: Wienner processes,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (22) where H =
Revised: (22) where H =

Original: ˆ H and B = ˆ B , assuming the learning rate and batch size is the same.
Revised: ˆ H and B = ˆ B , assuming the learning rate and batch size is the same.

Original: The proof follows directly from a theorem (Theorem 4) from (Patrini et al., 2017), stating that the curvature of the loss surface (Hessian) is invariant with respect to the noise when the neural network only uses piecewise linear function as its activation function.
Revised: The proof follows directly from a theorem (Theorem 4) from (Patrini et al., 2017), stating that the curvature of the loss surface (Hessian) is invariant with respect to the noise when the neural network only uses piecewise linear function as its activation function.

Original: 6.4. P OLYAK ’ S A VERAGE FOR THE W EIGHTS O PTIMIZATION
Revised: 6.4. P OLYAK ’ S A VERAGE FOR THE W EIGHTS O PTIMIZATION

Original: Polyak et.
Revised: Polyak et.

Original: al (Polyak & Juditsky, 1992) proved that the iterate average gives the optimal convergence rate and approximates the optimal by using the average of the iterates online as
Revised: al (Polyak & Juditsky, 1992) proved that the iterate average gives the optimal convergence rate and approximates the optimal by using the average of the iterates online as

Original: [Equation]
Revised: [Equation]

Original: and the iterate average after T step is
Revised: and the iterate average after T step is

Original: [Equation]
Revised: [Equation]

Original: where θ L and θ G indicate the weights of the leaner and guider respectively.
Revised: where θ L and θ G indicate the weights of the leaner and guider respectively.

Original: Next we prove the theorem 12 that in the presence of the noisy labels, when we update the weights of the learner model, the distance between the learner and the guider suggests the correctness of the labels assuming the invariance of the Hessian of the learner and guider by the lemma.
Revised: Next we prove the theorem 12 that in the presence of the noisy labels, when we update the weights of the learner model, the distance between the learner and the guider suggests the correctness of the labels assuming the invariance of the Hessian of the learner and guider by the lemma.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: In order to show Eq. 13, it is equivalent to show
Revised: In order to show Eq. 13, it is equivalent to show

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We denote
Revised: We denote

Original: [Equation]
Revised: [Equation]

Original: These are two stochastic optimization process described above for the stochastic gradient descent and the iterate average.
Revised: These are two stochastic optimization process described above for the stochastic gradient descent and the iterate average.

Original: ¯ θ S and ¯ θ T can be seen as the deviation from the optimal point.
Revised: ¯ θ S and ¯ θ T can be seen as the deviation from the optimal point.

Original: Hence we need to show
Revised: Hence we need to show

Original: [Equation]
Revised: [Equation]

Original: The left hand side of Eq. 27 gives
Revised: The left hand side of Eq. 27 gives

Original: [Equation]
Revised: [Equation]

Original: In order to evaluate the right hand side of the equation
Revised: In order to evaluate the right hand side of the equation

Original: [Equation]
Revised: [Equation]

Original: we recall the Green’s function for the
Revised: we recall the Green’s function for the

Original: Ornstein-Uhlenbeck process,
Revised: Ornstein-Uhlenbeck process,

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: the right hand side of Eq.
Revised: the right hand side of Eq.

Original: 27 becomes,
Revised: 27 becomes,

Original: [Equation]
Revised: [Equation]

Original: where we assume that the Hessian is full rank and its inverse has an eigen decomposition
Revised: where we assume that the Hessian is full rank and its inverse has an eigen decomposition

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When
Revised: When

Original: [Equation]
Revised: [Equation]

Original: The Eq. 13 holds.
Revised: The Eq. 13 holds.

Original: If we make assumption as in Jastrzebski et al.
Revised: If we make assumption as in Jastrzebski et al.

Original: (2017), i.e., covariance of the local minimal is approximated by the Hessian, namely, H = Σ .
Revised: (2017), i.e., covariance of the local minimal is approximated by the Hessian, namely, H = Σ .

Original: We observe that the last equation in the proof above becomes
Revised: We observe that the last equation in the proof above becomes

Original: [Equation]
Revised: [Equation]

Original: where L is the Laplacian.
Revised: where L is the Laplacian.

Original: This indicates that our approach favors the loss surface around minimal of a lower mean curvature and hence a narrower minimal.
Revised: This indicates that our approach favors the loss surface around minimal of a lower mean curvature and hence a narrower minimal.