I TERATIVELY L EARNING FROM THE B EST
Anonymous authors
Paper under double-blind review
A BSTRACT
We study a simple generic framework to address the issue of bad training data; both bad labels in supervised problems, and bad samples in unsupervised ones.
Our approach starts by ﬁtting a model to the whole training dataset, but then iteratively improves it by alternating between (a) revisiting the training data to select samples with lowest current loss, and (b) re-training the model on only these selected samples.
It can be applied to any existing model training setting which provides a loss measure for samples, and a way to reﬁt on new ones.
We show the merit of this approach in both theory and practice.
We ﬁrst prove statistical con
- sistency, and linear convergence to the ground truth and global optimum, for two simpler model settings: mixed linear regression and gaussian mixture models.
We then demonstrate its success empirically in a) saving the accuracy of existing deep image classiﬁers when there are errors in the labels of training images
, and
(b
) improving the quality of samples generated by existing DC-GAN models, when it is given training data that contains a fraction of the images from a different and unintended dataset.
The experimental results show signiﬁcant improvement over the baseline methods that ignore the existence of bad labels/samples
I NTRODUCTION
This paper is motivated by the problem that training large machine learning models requires wellcurated training sets with lots of samples.
This is an issue in both supervised and unsupervised settings.
For supervised problems (e.g. multi-label classiﬁcation) we need very many properly labeled examples; generating these requires a lot of human effort.
Errors in labels can signiﬁcantly affect the accuracy of naive training.
For example, a faulty CIFAR-10 dataset with 20% of automobile images mis-labeled as “airplane” (and so on for the other classes) leads to the accuracy of a neural architecture like WRN of Zagoruyko & Komodakis (2016) to go from over 90% to about 70%.
Even in unsupervised problems, where there are no labels, spurious samples are a nuisance.
Consider for example the task of training a generative adversarial network (GAN) Goodfellow et al.
(2014) to produce realistic images of faces.
While this has been shown to work well using a dataset of face images 1 , like Celeb-A Liu et al.
, it degrades quickly if the training dataset is corrupted with non-face images.
Our experiments illustrate these issues.
Main idea We study a simple framework that addresses the problem of spurious data in both supervised and unsupervised settings.
Note that if we knew which samples were clean, we could learn a model; conversely, if we had a good model, we could ﬁnd the clean samples.
But we do not have either; instead we have a chicken-and-egg problem: ﬁnding the best subset of samples needs a model, but ﬁnding a model needs a subset of samples.
In this paper, we take the natural approach to this problem by (0) ﬁtting an initial model on all samples, and then alternating between (1) ﬁnding the best subset of samples given a model, and (2) ﬁtting a model given a subset.
2 Our approach is most related to the works of Bhatia et al.
and Vainsencher et al.
and we contrast to these in the related work section.
Our approach can be applied to any existing model that provides (a) conﬁdence scores for samples, and (b) a way to train the model for a set of samples.
Indeed in our experiments we use existing models and training code, but with iterative (re)selections of samples as above.
In supervised settings, any model that maps features to responses typically associates a conﬁdence to every sample, e.g., in linear regression, given a model θ , the conﬁdence in a sample ( x i , y i ) increases as its squared error ( y i − x (cid:62) i θ ) 2 decreases.
For a neural network classiﬁer, on the other hand, the conﬁdence can be measured by the cross entropy loss between the output probability distribution and the true distribution.
In unsupervised settings, the conﬁdence of a sample is given by its “likelihood” under the model.
If we are lucky, e.g., in Gaussian mixture models, this likelihood is explicit and can be evaluated.
However, if there is no explicit likelihood, we need to rely on a surrogate.
E.g., in our
GAN setting below, we rely on the output of the discriminator as a conﬁdence measure.
We also note an important caveat: this paper is not focused on (and may not be effective for) the settings of adversarial examples, or of gross corruptions of features.
Rather, our focus is to address the issue of bad labeling or curation of data, as motivated above.
O UR R ESULTS AND P APER O UTLINE
We show that our framework (outlined in Section 3) can address the spurious sample problem, both theoretically and empirically and for both supervised and unsupervised problems:
(1) Mixed linear regression (Section 4) and
Gaussian mixture models
(Section
) : For each of these mixture problem settings, under the standard statistical assumptions, we establish both statistical consistency and linear convergence of our iteratively updated model parameters This holds
true from any initial point; the convergence is to
the mixture component that is closest to the initial point.
(2) Neural network classiﬁers (Section 6)
: We investigate image classiﬁcation in CIFAR-100,
CIFAR-10 and MNIST datasets (each with a different and appropriate existing neural architecture) in the setting where there are systematic errors in the labels – i.e. every bad example in one label class comes from the same other label class.
We show that our framework can signiﬁcantly improve the classiﬁcation accuracy over the baseline method that ignores the existence of bad samples.
(3) Deep generative models (Section 7): We show both quantitatively and qualitatively the improvement of using our algorithm for image generation when the dataset consists of two types of images.
Speciﬁcally, investigate training a GAN to (a) generate face images as in the “intended”
Celeb-A dataset, but when the given training data has a fraction of the samples being the “spurious”
CIFAR-10 images, and (b) generate intended MNIST digits when training data contains a fraction of samples from the spurious Fashion-MNIST dataset.
A crucial innovation here is that the output of the discriminator network in the GAN can be used as a conﬁdence metric for the training samples.
In Section 8, we discuss the positives and negatives of our results, underlying intuition and also some future directions.
All of our experiments were done on clearly identiﬁed publicly available neural network architectures, datasets, and training software.
We also clearly describe our error and noise effects; we expect them to be easily reproducible.
R ELATED W ORK
Alternating minimization for robustness Two works are most related to ours.
Bhatia et al.
(2015) proposes an iterative hard thresholding algorithm for adversarial noise in linear regression responses; they establish statistical guarantees and linear convergence under this harder noise model.
We focus on multiple mixed linear regression, a different setting that results in an easier noise model, and we provide convergence guarantee for recovering any single mixture component starting from a local region.
In addition we also prove these for Gaussian mixture models, which is not studied in their work.
Vainsencher et al.
(2017) proposes what is essentially a soft version of our method, and proves local convergence and generalization.
However they do not have any initialization, and hence no global or consistency guarantees.
Neither of these empirically explore the overall approach for more complex / neural network models.
Robust regression Recent work on robust regression consider strong robustness in terms of both the inputs and the outputs and provide guarantees for constant corruption ratio Diakonikolas et al.
; Prasad et al.
; Klivans et al.
Chen et al.
; Balakrishnan et al. (2017a);
Liu et al.
(2018) study the high dimensional setting under this regime.
These algorithms usually require much more computation, compared with methods for dealing with noisy outputs, e.g., Bhatia et al.
Another type of robustness considers heteroscedastic noise for every sample.
From the learning perspective, Anava & Mannor (2016); Chaudhuri et al.
both require strong constraints on the noise model which largely reduces the degree of freedom.
Anava & Mannor (2016) considers the online learning setting, while Chaudhuri et al.
(2017) considers the active learning setting.
Mixed linear regression Alternating minimization type algorithms are used for mixed linear regression with convergence guarantee Yi et al.
Chen et al.
(2014) provides a convex formulation for the problem.
Balakrishnan et al. (2017b) shows expectation-maximization (EM) algorithm is provably effective for a set of tasks including the basic setting of mixed linear regression and Gaussian mixture model.
Sedghi et al. (2016) extends to the multiple component setting, where provable tensor decomposition methods are used.
More recently, Li & Liang (2018) gives a more general result for learning mixtures of linear regression.
On the other hand, Ray et al. (2018) studies the problem of ﬁnding a single component in mixed linear regression problem using side information.
Noisy labels Classiﬁcation tasks with noisy labels are also of wide interest.
Fr ´ enay et al.
(2014) gives an overview of the related methods.
Theoretical guarantee for noisy binary classiﬁcation has been studied under different settings Scott et al.
; Natarajan et al.
; Menon et al.
More recently, noisy label problems have been studied for deep neural network based models.
Reed et al.
and Malach & Shalev-Shwartz (2017) develop the idea of bootstrapping and query-bycommittee into neural network based models.
On the other hand, Khetan et al.
and Zhang &
Sabuncu (2018) provide new losses for training under the noise.
Sukhbaatar & Fergus (2014) adds a noise layer into the training process, while Ren et al.
(2018b) provides a meta-algorithm for learning the weights of all samples by referencing to a clean subset of validation data during training. Ren et al. (2018a) consider building robust classiﬁers for the positive-unlabeled classiﬁcation problem using l 0 norm penalization.
Others Bora et al. (2018) studies the problem of noisy images for GANs.
In their setting, all images are of low quality under some measurements, while in our problem, we consider the image dataset consists of (possibly more than) two types of images, which is different.
There are also studies that consider noisy samples more generally.
The classical RANSAC method Fischler &
Bolles (1981) provides a paradigm for ﬁtting a model to experimental data, and can be potentially used for the noisy data setting.
However, the algorithm requires multiple times of random sampling to ﬁnd a valid consensus score, which is conceptually different from our idea of doing iterative ﬁltering.
EM algorithm Moon (1996) can also be used for detecting noisy samples, e.g., in the setting of mixture models Balakrishnan et al. (2017b).
However, our algorithm does not need to know the likelihood for the bad samples for the iterative updates.
I TERATIVELY L EARNING FROM THE B EST ( ILFB )
We now describe the setup and algorithm, in a uniﬁed way that covers both supervised and unsupervised settings.
We will then specialize to the many speciﬁc problems we study.
We are provided with samples s 1 , · · · , s n , using which we want to train a model parameterized by θ ∈ R d .
We are also provided with a loss function; let f θ ( s ) denote the loss of a sample s when the parameters are θ .
Let τ ∈
[0 , 1] be the fraction of data we want to ﬁt.
With this setup, the idea of learning from the best can be viewed as trying to solve
[Equation]
We do this by alternating between ﬁnding S and ﬁnding θ , as described below in Alg. 1.
1: input: samples { s i } ni =1 , fraction τ , max. iteration number T 2: (optional) initialize: set t ← 0 and initially ﬁt all samples θ 0 ← arg min θ (cid:80) i ∈ [ n ] f θ ( s i ) , 3: while t < T do 4: calculate current losses f θ t ( s i ) for all samples s i , i ∈ [ n ] 5: ﬁnd new best set of samples
S t ← arg min S : | S | = τn (cid:88) i ∈
S
f θ t ( s i ) by sorting them according to losses f θ t
( s i ) and choosing the ones with smallest losses.
6: ﬁt a model for this new set θ t +1 ← arg min θ (cid:88) i ∈ S t f θ ( s i ) 7: t ← t +8: return: θ T
Comments Steps 4, 5 and 6 above can be done approximately, when exact solutions are unavailable.
For example, in our experiments involving deep neural networks (both the classiﬁers and the generative models), for step 6 we do not ﬁt a model exactly, but rather do training using stochastic gradient descent.
Also, for our experiments on training GANs on the best subset, we do not have an explicit loss; we instead use the loss at the output of the discriminator as a surrogate (more details on that below).
Similarly, the initialization step above can possibly be replaced with some problem-setting dependent alternative, if one is available.
In the following, we describe how this directly specializes to each of our four settings.
ILFB FOR M IXED L INEAR R EGRESSION
We ﬁrst describe ILFB as specialized to the case when the θ represents the parameters of a linear regression, and then describe the statistical setting of mixed linear regression where we provide rigorous statistical guarantees on its performance.
Algorithm The samples are s i =
( x i , y i ) with features x i ∈ R d and responses y i ∈ R , and θ ∈ R
d are the parameters of the linear model.
The loss is f θ ( x , y ) =
( y − x (cid:62) θ ) 2 .
The algorithm thus initializes θ 0 by doing an ordinary least squares (OLS) on all the samples, and then alternates between ﬁnding the set of τn samples with the lowest squared error for the current θ , and then ﬁnding the new θ by doing an OLS on this set.
Statistical Setting In the standard and widely studied mixed linear regression problem there are n samples ( x i , y i ) , with features x i ∼ N (0 , I d ) being standard gaussians in d dimensions.
The full set of samples S =
[ n ] can be splitted into m sets S = ∪ j ∈ [ m ] S j , each corresponding to samples that generate from one mixture component, | S j | [Equation] τ [Equation]
n . The response variables y i are given by:
[Equation]
where (cid:15) i ∼ N (0 , σ 2 ) is the additive noise, and θ ( j ) s are assumed to be unit norm vectors.
We analyze our algorithm in the fresh sample setting, where iteration t ﬁnds θ t +1 from θ t using a new set of samples.
This too is standard in the statistical analysis of mixed linear regression, and corresponds to the case of using mini-batches of samples to learn a model.
We now prove two results: the ﬁrst shows that the iterates converge linearly to some θ ( j ) once θ t is closer enough to θ ( j ) than any of the other components, and the second shows that the simple initialization step ensures a θ 0 that is close enough to the dominant component θ ( j ) under mild assumptions.
Theorem 1 ( local linear convergence).
For the mixed linear regression setting as above, we run
ILFB as shown in Algorithm 1.
Suppose for some j ∈ [ m ] , the iterate θ t and τ satisfy: (C1) τ ≤ 0 . 99 τ (cid:63)j ; (C2) (cid:107) θ t − θ ( j ) (cid:107) 2 ≤ C ( τ ) min l ∈ [ m ] \{ j } (cid:107) θ t − θ ( l ) (cid:107) 2 − (cid:112) 1 − C 2 ( τ ) σ , where C ( τ ) =
min { c 1 c 3 τ 1 − τ , 1 } . Then, the next iterate θ t +1 of the algorithm satisﬁes
[Equation]
with high probability (i.e., with probability 1 − n − c 0 ), provided the number of samples n ≥ c d log d c 22 τ , where c 1 ∈ (0 , 1) , c, c 0 , c 2 , c 3 are constants.
Theorem 1 establishes a linear dependence of the sample complexity on the dimension d .
The required sample size increases as τ gets smaller.
For τ large enough such that C ( τ ) = 1 , the condition on θ t does not depend on σ .
However, notice that for smaller τ , the condition of θ t tolerates smaller noise. This is because, even if θ t is very close to θ ( j ) , when the noise and the density of samples from other mixture components are both high, the number of samples from other components selected by the current θ t would still be quite large, and the update will not converge to θ ( j ) .
The proof for the theorem is in Appendix C.1, and synthetic experiments verifying the performance of ILFB are in Appendix C.2. In Appendix C.2, we show that our guarantee for the algorithm is tight, in the sense that there always exist a σ dependency even if we take inﬁnite samples.
In the noiselss setting, our theorem guarantees local linear convergence with exact recovery.
ILFB FOR G AUSSIAN M IXTURE M ODEL
In this section, we describe ILFB applied to the setting of Gaussian mixture model, and provide statistical guarantees on its performance.
Algorithm The samples s i =
x i are points in R d ,
θ ∈ R
d is the mean we want to estimate, and f θ ( x ) is proportional to the negative log-likelihood, i.e., f θ ( x ) = (cid:107) x − θ (cid:107) 22 .
The algorithm thus initializes the mean by ﬁnding the average of all n points, and then iteratively ﬁnds the set of τn points closest to the current mean, and updating the mean to be the average of this set.
Statistical setting Given a dataset D
= { x i | i ∈ S
= { 1 , · · · , n }} , where S can be splitted into disjoint sets [Equation] = [Equation]
The samples are generated following:
[Equation]
We now prove two results: the ﬁrst shows that iterates converge linearly under the fresh sample setting in a local region, while the second shows with some additional conditions, that the initialization of ILFB falls into the local region of the dominant component. Theorem 2 ( local linear convergence). We run Alg. 1 on the given dataset D generated following (2).
Suppose for some j ∈ [ m ] , the iterate θ t and parameter τ in Algorithm 1 satisfy: (C1) τ ≤ 0 . 99 τ (cid:63)j ; (C2) (cid:107) θ t − θ ( j ) (cid:107) 2 ≤ 0 .
9 min l ∈ [ m ] \{ j } (cid:107) θ t − θ ( l ) (cid:107) 2 − c 3 σ (cid:113) log 2(1 − τ ) τ [Equation] then, with high probability (i.e., with probability 1 − n − c 0 ),
[Equation]
where c 1 ∈ (0 , 1) , c 0 , c 2 , c 3 are constants.
[Table] Here, the 0 . 9 factor in the assumption is chosen for convenience of the analysis.
Essentially, what we need is this value being less than one.
For τ < 12 , we require an additional separation term that depends on σ .
This term essentially implies a √ log kσ separation distance (consider all mixture components have weights 1 /k ), which is the lower bound for computationally efﬁcient algorithm to learn a Gaussian mixture model according to Regev & Vijayaraghavan (2017).
Intuitively, given the required initialization condition, the samples selected by the initialization parameter contain more than half of the good samples, and we can show that the algorithm is guaranteed to converge.
Notice that there is no dependency between n and d .
In the extreme case where σ = 0 , we only need n to be a constant value: from any good initialization point, the algorithm will converge to θ ( j ) in one step.
The proof for the theorem is in Appendix D.1, and synthetic experiments verifying the performance of ILFB are in Appendix D.2.
Again, in Appendix D.2, we show that the our guarantee of the algorithm is tight.
ILFB FOR D EEP I MAGE C LASSIFICATION WITH S YSTEMATIC T RAINING L ABEL E RROR
Experimental setting We consider systematic errors in labels , i.e. the setting where all the samples that actually come from a class “a” are given the same bad label “b”, a setting that is less benign than one with more randomness in the errors.
We investigated the ability of our framework to account for these errors for the following dataset – neural architecture pairs:
(a) MNIST LeCun et al. (1998) dataset with a standard 2-layer CNN(b)
CIFAR-10 Krizhevsky & Hinton (2009) with a 16-layer
WideResNet 4 Zagoruyko & Komodakis (2016) (c) CIFAR-100 5 with a 28-layer WideResNet
In each of these cases, the neural network has excellent performance when there are no label errors in training; this degrades signiﬁcantly as the fraction of spurious samples increases.
Algorithm
In each case, the algorithm involves ﬁrst training the NN on all the samples, and then iteratively alternating between (a) picking the fraction τ of samples in each label class with the lowest cross-entropy loss under the current θ , and (b) retraining the model from scratch with these picked samples, to get a new θ .
Training was done on MXNet.
Mapping back to the development in Section 3, here θ represents the weights of the neural network, and f θ ( s ) is the standard crossentropy loss used in training.
Further details When training on all the samples, we run simple stochastic gradient descent algorithm with initial learning rate 0.5/0.1/0.1 and batch size 1000/256/64 for MNIST/CIFAR10/CIFAR-100, respectively.
The learning rate is divided by 5 at the 50 -th epoch and each ex- [Table] periment runs for 80 epochs.
For each iteration of update using our ILFB algorithm, since less samples are used at each epoch, we adjust the total epoch number accordingly, so that the training for every iteration uses the same number of SGD updates.
We set τ to be 5% less than the true ratio of “good” samples.
Results Table 1 shows the results for the baseline, oracle and our methods.
Please see the caption thereof.
We observe signiﬁcant improvement over the baseline under all experiments.
In Appendix
B, we provide a comparison of using different initialization methods, where ILFB performs better than its counterpart with random initialization.
In Appendix E, we provide more experimental results illustrating that ILFB is not sensitive to mis-speciﬁed τ , the improvement of this iterative procedure compared with a single iteration, the advantage over other outlier removal options, and its capability of handling random label noise.
ILFB FOR D EEP G ENERATIVE M ODELS WITH M IXED T RAINING D ATA
Experimental setting We consider training a GAN – speciﬁcally, the DC-GAN architecture Radford et al.
to generate images similar to those from a good dataset, but when the training data given to it contains some fraction of the samples from a different bad dataset.
All images are unlabeled, and we do not know which training sample comes from which dataset.
We investigated the efﬁcacy of our approach in three such settings:
(a) When the good dataset is the Celeb-A face images, and the bad dataset is CIFAR-10.
(b) When the good dataset is the CIFAR-10 face images, and the bad dataset is Celeb-A. (c) When the good dataset is MNIST digits, and the bad is Fashion-MNIST Xiao et al.
For each of these,
we
consider different fractions of bad samples in the training data, evaluate their effect on standard GAN training, and then the efﬁcacy of our approach as we execute it for uptoiterations.
Algorithm Recall that training a GAN consists of updating the weights of both a generator network and a discriminator network; our model θ is the parameters of both of these networks.
Unlike the classiﬁcation seeing, we now need to develop a loss f θ ( s ) – and unlike in the simple Gaussian mixture model setting we do not have explicit access to a likelihood function for the generative model.
Our crucial innovation is to use the loss 7 at the output of the discriminator – the same one used to train the discriminator – as the f θ ( s ) .
The algorithm starts by training on all samples, and then alternates between picking training samples with the smallest discriminator loss, and retraining [Figure] [Figure] the model on these picked samples.
Here training means updating both the generator and discriminator network weights, which is done via SGD.
Again, we set τ to be 5% less than the true ratio of “good” samples. [Figure]
Results We evaluate the effect of bad training samples, and the improvement by our approach, in two ways: quantitatively in Table 2 and qualitatively in Figures 1 and 2.
Figure 3 illustrates a failure case, when the bad set is too big, which we feel gives insight into what is going on.
Please see the respective captions.
The failure example happens for MNIST-Fashion dataset when ratio of MNIST images is 60% .
In fact, for every iteration, ILFB selects all images from FashionMNIST (which counts for 0 .
4 n ) and 0 .
15 n MNIST images (the least number of MNIST images the algorithm can select).
See Fig.
3 for qualitative evaluation of the results.
In Appendix E, we provide more experimental results illustrating that ILFB is not sensitive to mis-speciﬁed τ and the advantage over other outlier removal options.
D ISCUSSION
We demonstrated, both theoretically for simpler settings, and empirically for more challenging neural network ones, the merit of iteratively ﬁtting to the best samples – in both supervised and unsupervised problems.
The underlying intuition is that ILFB can focus in on the “good” samples, provided it is properly initailized; and when the number of bad samples is signiﬁcant but not overwhelming, initial ﬁtting on all the data is good enough to get started.
We now add several discussions on the merits and otherwise of our results:
(1) One way to view the effect of ILFB in the GAN setting: a discriminator is likely least certain on the smallest modes of a distribution, and our process of dropping training samples based on this results in the elimination of smaller modes, which correspond to modes from the bad samples.
This is illustrated by the failure case in Fig.
3, where the presence of a high fraction of ”bad” fashionMNIST samples makes their modes comparable to those of the ”good” MNIST.
(2) Retaining samples best ﬁt by the current model is inherently local, in the sense that samples with large errors are ignored.
Finding a good initialization to start with is thus important; without this we may have bad performance.
When the size of the good samples is comparatively large, an initial ﬁt on all the samples is good enough; otherwise, we may use other alternatives to initialize the model, e.g., ﬁt the model on a smaller dataset with clean data. (3) The ILFB algorithm is simple and efﬁcient enough to be applied to most modern machine learning tasks. When there exists extra computation power or knowledge on the speciﬁc dataset, it is certainly possible to improve based on ILFB . However, we believe ILFB is simple and general enough, and may serve as a strong result when more complicated outlier detection or noisy label algorithms are proposed.
A E FFECT OF MIS - SPECIFICATION A. M IS - SPECIFICATION FOR A CLEAN DATASET
We test the performance of ILFB under mis-speciﬁed τ when the dataset is clean, i [Figure]
e., there is only a single component.
We choose random θ (cid:63) ∈ R d with unit norm, the mis-speciﬁed τ is set as 0 .
95 , dimension d = 100 , and sample size n ∈ { 500 , 1000 , 1500 } , T = 30 .
As shown in Fig.
4, as σ gets larger, the gap between ILFB and OLS estimator (in terms of (cid:96) 2 distance) increases linearly
[Figure] [Figure]
We further show the performance of ILFB as sample size increases.
In Fig.
6, the asymptotic performance under different noise levels is shown.
For the clean dataset, ILFB is observed to give consistent estimate.
We observe similar performance in the Gaussian setting as well.
[Figure]
A. M IS - SPECIFICATION FOR A NOISY DATASET
In this part, we test the performance when using different τ values for a noisy dataset.
Again, we present the result for the mixed linear regression setting.
We run ILFB for 10 iterations and the results are based on an average of 50 random runs.
In Fig.
5, we see that the performance is not sensitive to the mis-speciﬁed τ in a large region.
Interestingly, ILFB still performs well when τ is slightly larger than the true ratio of the component we are interested in.
B R ANDOM I NITIALIZATION FOR C LASSIFICATION
We study the effect of initialization for the classiﬁcation problem using MNIST (5%) dataset.
In
Table 3, we present the accuracy after 5 -th iteration of training for both random initialization and the original ILFB methods.
Even with random initialization, the performance is consistently better than the baseline, however, the improvement of accuracy is smaller when compared to the original
ILFB , which uses all samples for training as initialization.
[Table]
C L INEAR R EGRESSION M ODEL
Since we focus on recovering the parameters of one mixture component from the mixed linear regression model, we refer to the targeted component as the ‘good’ component, and the rest are ‘bad’ components. More speciﬁcally, the full set of samples S = [ n ] can be splitted into a set of “good” samples S good , and m sets of “bad” samples ∪ j ∈ [ m ] S j bad .
The response variables y i are given by:
[Equation]
where (cid:15) i ∼ N (0 , σ 2 ) is the additive noise, and θ (cid:63) as well as all the θ ( j ) are assumed to be unit norm vectors.
For simplicity of the notation, we present the proof by assuming y i for bad samples have the form y i = n i + e i , where n i corresponds to x (cid:62) i θ ( j ) , such that i ∈
S
j bad .
Let θ t be the current learned parameter.
θ t +1 is learned by using one iteration in Algorithm 1.
Namely, we ﬁrst select a subset S t of size τn with the smallest loss ( y i − x (cid:62) i θ t ) 2 , then re-calculate the MLE estimator only using those samples.
Denote W t as the diagonal matrix whose diagonal entry W ii equals 1 when the i -th sample is in set S t , otherwise equals 0 .
Then,
[Equation]
where we used the fact that W 2 t = W t .
Notice that W (cid:63) is the ground truth for W t (accordingly, deﬁne S (cid:63) for the ground truth of S t ).
For clearness of the presentation, we ignore the subscript t when there is no ambiguation.
Then, according to the deﬁnition of y and reorganizing the terms, (3) can be expanded as
[Equation]
where as we addressed at the beginning of this section
[Equation]
for vector n , given i ∈ S bad , n i = x (cid:62) i θ ( j ) for some j ∈ [ m ] .
Therefore,
[Equation]
We bound the three terms 4) separately [Equation]
Helpful lemmas
Lemma 3 (Sub-Gamma property for the q -quantile) .
Let n ≥ 3 , let X (1) ≥ · · · ≥ X ( n ) be the order statistics of the absolute value of standard Gaussian samples.
qn is an integer for some constant q ∈ (0 , 1) .
Let v n =
[Equation]
This shows the sub-gamma property of X ( qn ) , and as a result, for all t > 0 , and constant 12 < q < 1 ,
[Equation]
This result is generalized from the result in Boucheron et al.
Lemma 4.
Suppose we have two Gaussian distributions D 1 = N (0 , ∆ 2 ) , D 2 = N (0 , 1) .
We have fn i.i.d.
samples from D 1 and (1 − f ) n i.i.d.
samples from D 2 .
Denote the set of the top ¯ fn samples with smallest abstract values as S ¯ fn , where ¯ f ≤ f . Then, with high probability, for ∆ ≤ 1 , at most (cid:0) c [Equation]
[Equation] − f ) n [Equation] log n, log n (cid:9)(cid:1) samples in S ¯ fn are from D 2 .
Proof of Lemma 4.
First, let us consider the probability of a random variable from D 2 has smaller abstract value than a random variable from D 1 . This would give us a rough idea about the occupancy of the samples from D 2 in the top set. Denote the two variables from D 1 , D 2 as u 1 , u 2 respectively. Then, with basic calculations, we have
[Equation] This calculation implies that if given same number of samples from D 1 and D 2 , it is reasonable to expect that only around 2∆ fraction of the top set samples are from D 2 , when ∆ is small. In the original setting where we have ¯ fn samples from D 1 and (cid:0) 1 − ¯ f (cid:1) n from D 2 , we should expect around 2∆ 1 − ¯ f ¯ f ¯ fn = 2∆ (cid:0) 1 − ¯ f (cid:1) samples in the top set are from D 2 . Now, we aim for a rigorous proof of the result with good concentration.
Our idea is to ﬁrst show concentration result for the maximum value from distribution D 1 , and then, we show the high probability upper bound for the number of samples from D 2 that are less than the maximum value. Step I – the sample with maximum abstract value from D 1 .
We know that for random normal i.i.d. Gaussian variables x i , i ∈ [ n ] ,
[Equation] Therefore, for fn samples from D 1 , with high probability, the maximum abstract value is in the order of O ( √ log n ∆) . Step II. On the other hand, for a random u 2 ∼ D 2 , we know that for small positive values
δ = c √ log n ∆ , P [ | u 2 | ≤ δ ] ≤ (cid:113) 2 π δ gives a tight upper bound.
Let M δ,i be the event sample
u i from
D 2 has abstract value less than δ , and a Bernoulli random variable m i,δ that is the indicator of event
M δ,i holds or not.
Then,
[Equation]
For independent random variable x i s, i ∈ [˜ n ] that lie in interval [0 , 1] , with X = (cid:80) i x i and µ = E [ X ] , Chernoff’s inequality tells us [Equation]
As a consequence, for m i,δ s, we have
[Equation] For the ﬁrst case, where γ ∈ [0 , 1] , we can set γ = c (cid:113) log n δ (1 − f ) n to get high probability guarantee. The constraint on γ requires δn > c log n for some ﬁxed c . On the contrary, when this is violated, i.e., when δ is much smaller, then, by the Chernoff bound for the case γ > 1 , we can set γ = c log n δ (1 − f ) n .
Combining Step I and Step II.
To summarize, for some ﬁxed constant c , with high probability:
• For ∆ > c √ log n n , at most 2 cδ (1 − f ) n = c √ log n ∆ (1 − f ) n samples in S ¯ fn are from D 2 . • For ∆ ≤ c √ log n n , at most (1 + γ ) δ (1 − f ) n = c log n samples in S ¯ fn are from D 2 . (cid:4) Lemma 5.
Assume τ < τ (cid:63) ( τ/τ (cid:63) = c τ < 1 is a constant), with high probability, we have:
[Equation]
where c 0 , · · · , c m , c are constants.
Proof of Lemma 5.
We focus on ﬁnding concentration results for the smallest eigenvalue of
X (cid:62) WX .
Rewrite X (cid:62) WX as
[Equation]
where 1 [ · ] is the indicator function.
Note that the values { l i,t } ni =1 are deﬁned as:
[Equation]
More speciﬁcally, consider the model setting, we have:
[Equation]
For all i ∈ S (cid:63) , l i,t s have the same distribution as δµ , where µ is a χ 2 (1) random variable, and δ is the magnitude δ = (cid:107) θ (cid:63) − θ t (cid:107) 22 + σ 2 .
The bad samples i ∈ [ n ] \ S (cid:63) , for i ∈ S
j bad have the same distribution as ( σ 2 + (cid:107) θ ( j ) − θ t (cid:107) 22 ) µ .
Also, the assumption in θ t assumes that the magnitude of the good distribution is smaller.
Applying the results in Lemma 3 and Lemma 4, assuming the condition in Theorem 1 holds, for given θ t , θ (cid:63) , there exists a constant c 1 such that the separation value for deciding top-
τn
or not is lower bounded by cδ with high probability.
For a ﬁxed c 0 , we consider the following quantity:
[Equation]
By the geometry of the constraint, we have that
[Equation]
In other words, the lower bound we have for the smallest eigenvalue of the expectation of (9) monotonically increases with c , and can be substituted with another constant 2 c 0 (notice that previously, we have a lower bound on the separation value).
Therefore, considering the sub-exponential property, we get the concentration result based on the above expected value:
[Equation]
Similarly, since (cid:107) θ j − θ t (cid:107) ≤ 3 , we have a lower bound O ( | S t ∩ S j | ) for i ∈ S t ∩ S j as well, as long as the set | S t ∩ S
j | ≥ c log 2 n .
In any case, we have the minimum eigenvalue is lower bounded by cτn .
[Equation]
T 2 in (4)
T 2 can be bounded as follows:
[Equation]
≤
[Equation]
for | S t \ S (cid:63) | ≥ c log 2 n (when | S t \ S (cid:63) | < c log 2 n , the ﬁnal convergence result still holds true).
As a consequence,
T
[Equation]
[Equation]
T 3 in (4) For this part, we can reuse the result we proved for T 1 .
Also, notice that with high probability:
[Equation]
Putting things together Combining Lemma 5, (10), and (11), we have:
[Equation]
We require n ≥ cd log d c 24 τ . Also, according to Lemma 4, given (cid:107) θ t − θ (cid:63) (cid:107) 2 ≤ c 3 c τ 1 − τ min j ∈ [ m ] (cid:107) θ t − θ j (cid:107) 2 − (cid:114) 1 − (cid:16) c 3 cτ 1 − τ (cid:17) 2 σ (for c 3 ∈ (0 , 1) ), then with high probability, | S t \ S (cid:63) | ≤ c 3 c (cid:48) τn . Notice that for small τ , the noise should not be too large.
Otherwise, even if θ t is very close to θ (cid:63) , because of the noise and the high density of bad samples, | S t \ S (cid:63) | would still be quite large, and the update will not converge.
[Figure] We verify the performance of ILFB on mixed linear regression model via synthetic experiments. When constructing the dataset we consider a special case where all bad samples are generated from a single β j .
The parameters for both the good and bad components are generated randomly on a unit sphere.
First, in Fig. 7 (left) we test the performance when τ (cid:63) varies in the interval [0 . 5 , 0 . 8] with σ = 0 . We compare: (1) MLE :
All samples are treated as good samples to ﬁnd the best parameter; (2) Oracle : learn from the τ (cid:63) n good samples given oracle access; (3) Super-Oracle : learn from n samples given oracle access; (4) R-Init+ILFB : ILFB with random initialization; (5) M-Init+ILFB : ILFB using MLE as initialization.
The parameter recovery ( y -axis) is measured by (cid:107) θ alg − θ (cid:63) (cid:107) 2 .
We can see that ILFB with MLE as initialization performs better than the random initialization counterpart, which shows the effectiveness of taking good initailizations.
Next, we show experimentally the speed of convergence in Fig.(right) with multiple levels of measurement noise (in the small noise regime).
The plots show linear convergence of each update step, which matches the result in Theorem 1.
C.2. A SYMPTOTIC PERFORMANCE
Next, we check the performance of ILFB asymptotically.
Our theory implies that the ﬁnal recovery will get to a noise ball centered at the true parameter, but does not ensure the algorithm goes to the true parameter, even when sample size goes to inﬁnity.
In this section, we show experimentally that this noise ball guarantee is indeed tight, i.e., one should not expect ILFB to give exact recovery even in the inﬁnite sample case.
Our experimental setting is as follows: we consider mixed linear regression with two components, where 60% of the samples comes from the interested component, and all x i s follow isotropic normal
Gaussian distribution.
The parameters for the two components are set as unit norm, and orthogonal to each other, with d = 10 .
We run ILFB for 15 iterations with τ set as 5% less than that of the true ratio.
Number of samples varies from 100 to 51200 .
Standard deviation is calculated based onruns for each experiment.
Fig.
8 provides the comparison between ILFB and oracle performance under different noise regime, i.e., σ = 0 .
We observe the performance of ILFB is close to the oracle in the small noise regime, while when the noise is comparable to the signal, there exists a gap between ILFB and the oracle, as the result of ILFB falls within the noise ball and stops decreasing, while the oracle will go to the exact solution asymptotically.
Similar performance appears in the large noise regime, however, for n ≤ 51200 , the gap between the two seems not too far apart, which is due to the dominance of the noise.
[Figure] C. I NITIALIZATION FOR LINEAR REGRESSION MODEL Theorem 6 ( initialization).
Given the model described in (1), for some j ∈ [ m ] , deﬁne d jm := min l ∈ [ m ] \{ j } (cid:107) θ ( j ) − θ ( l ) (cid:107) 2 , if (cid:80) l ∈ [ m ] \{ j } (cid:107) θ ( j ) − θ ( l ) (cid:107) 2 | S l | ≤ nκ d jm holds for some κ . We have: [Equation]
with high probability, where θ 0 = θ OLS , n ≥ c max { κ 2 dσ 2 ( d jm ) 2 , mκ 2 log( mκ ) ( d jm )} .
The assumption in Theorem 6 requires the total sum of distances of the parameters for samples from other components should not be too large. For a ﬁxed d jm , if the distance for some θ ( l ) becomes larger, it may become harder for the OLS estimator to be in the local region close to θ ( j ) .
When the condition holds for some large κ , e.g., κ > 2 . 2(1 + 1 − τ c 1 c 3 τ ) and σ <
[Equation]
1 − τ ) 2 − c 21 c 23 τ 2 when C ( τ ) < 1 (or for some large τ , κ > 2 . 2 ), following Theorem 1, the algorithm converges to a local O ( σ ) ball around θ ( j ) .
Proof.
We write out the OLS estimator as follows:
[Equation]
Therefore, with high probability, [Equation] ≤
[Equation]
n
[Equation]
n
[Equation]
n
[Equation] [Equation]
c
[Equation]
n
[Equation]
σ
[Equation]
n (cid:33) (12) ≤ 1 .κ d m , for n ≥ c max { κ 2 dσ 2[-( κ − 2) 
-] d 2 m , mκ 2 log( mκ ) d 2 m } , where in (12), we use the assumption (cid:80) j ∈ [ m ] (cid:107) θ (cid:63) − θ ( j ) (cid:107) 2 | S j bad | ≤ nκ min j ∈ [ m ] (cid:107) θ (cid:63) − θ ( j ) (cid:107) 2 .
(cid:4)
D G AUSSIAN M IXTURE M ODEL
Similar to Section C, we refer to the targeted component as the ‘good’ component and the rest are ‘bad’ components.
More speiﬁcally, S can be splitted into disjoint sets S
= S good ∪ S 1bad ∪· · ·∪ S
m bad , the samples follows:
[Equation]
Similar to the proof for linear regression setting, we use S t to denote the set of selected samples according to parameter θ t .
Then,
[Equation]
Assume S (cid:63) ⊂ S good to be the set of good samples that are closest to θ (cid:63) , and | S (cid:63) | = | S t | = τn .
In other words, S (cid:63) is selected by growing a ball centered at θ (cid:63) , until τn of the good samples fall into this ball, and we select them to be the set S (cid:63) .
Accordingly, let
[Equation]
With high probability, (cid:107) ¯ θ − θ (cid:63) (cid:107) 2 ≤ c √ dσ log n √ τn
= o ( σ √ d ) . For i ∈ S bad ∩ S t , the norm of x i − θ t is small.
More speciﬁcally,
[Equation]
(cid:107) 2 , ∀ i ∈ S bad ∩ S t , ∀ l ∈ S (cid:63) \ S t . For any good sample with index l , the value (cid:107) x
l − θ t (cid:107) 2 ≤ (cid:107) x l − θ (cid:63) (cid:107) 2 + (cid:107) θ (cid:63) − θ t (cid:107) 2 by triangle inequality, where the square of 1 σ (cid:107) x l − θ (cid:63) (cid:107) 2 follows χ 2 ( d ) distribution.
Denote B d ( q ) be the qquantile value of χ 2 ( d ) distribution, with high probability, the q -quantile of 1 σ (cid:107) x l − θ (cid:63) (cid:107) 2 for all good samples is bounded by 2 (cid:112) B d ( q ) . Next, we have
[Equation] which is based on the deﬁnition in (13) and (14).
Then,
[Equation] [Table] [Equation]
(17) (15) makes an expansion for each term: the norm of x i − θ t for i ∈ S t \ S (cid:63) should be small because of the selection rule, while the norm of x i − θ (cid:63) for i ∈ S (cid:63) \ S t should be small because they are good samples. B d (cid:16) | S (cid:63) ∩ S t | τ (cid:63) n (cid:17)
= O ( d ) .
Inequality (16) splits (15) into 4 terms.
The second term in (16) is bounded based on triangle inequality and the fact that if a sequence is element-wise larger than the other sequence, then the order statistics is also larger. For the second last term in (17), since the set
S (cid:63) is selected by spanning a ball centered at θ (cid:63) , the distance to the center from any sample in set S (cid:63) is bounded by cσ √ d for some constant c . Therefore,
[Equation]
In order for (18) to converge, we expect | S (cid:63) \ S t | < τn/ 2 . For τ (cid:63) ≥ 12 , given (cid:107) θ t − θ (cid:63) (cid:107) 2 ≤ 0 . 9 min j ∈ [ m ] (cid:107) θ t − θ j (cid:107) 2 , there exists c 1 ∈ (0 , 1) , such that | S (cid:63) \ S t | ≤ c 1 2 τn with high probability (i.e., more than half of samples in S t are from the correct component).
As a consequence,
[Equation]
For τ < 12 , in order to guarantee | S (cid:63) \ S t | < τn/ 2 , we not only require (cid:107) θ t − θ (cid:63) (cid:107) 2 ≤ 0 . 9 min j ∈ [ m ] (cid:107) θ t − θ j (cid:107) 2 .
This is because, the samples generated from θ (cid:63) may be a small fraction of the whole dataset (less than half), and when θ t is slighly closer to θ (cid:63) , it will still collect more bad samples because of the higher density.
Intuitively, when σ is large enough, for any θ t ∈ R d , | S (cid:63) \ S t | can be larger than τn/ 2 .
Observe that the median of squared distance from the good samples to θ t (denoted as med good ) concentrates at the sample mean, which concentrates at the expected mean, which is (cid:107) θ t − θ {+(cid:63) (cid:107) 
+}
+ σ 2 d .
On the other hand, we require the number of bad samples whose squared distance to θ t is less than med good is less than τn/ 2 , which corresponds to the τ 2(1 − τ ) -quantile value to be greater than med good .
Therefore, a valid θ t should satisfy:
[Equation]
where n ∼ N (0 , I d ) is a normal Gaussian vector. By sub-exponential inequality, and reorganizing the terms, a sufﬁcient condition we require is (cid:107) θ t − θ (cid:63) (cid:107) 2 ≤ min j ∈ [ m ] (cid:107) θ t − θ j (cid:107) 2 − c (cid:113) log 2(1 − τ ) τ σ .
This condition essentially implies a √ log kσ separation distance, which is the lower bound for computationally efﬁcient algorithm to learn a Gaussian mixture model according to Regev & Vijayaraghavan (2017).
D. S IMULATION RESULTS
In this part, we provide synthetic experimental results for the Gaussian mixture model.
Most of the settings are similar to the settings for mixed linear regression in Section C.2.
For clearness, we will re-state some of the deﬁnitions/settings.
We verify the performance of ILFB for Gaussian mixture model through synthetic experiments.
In Gaussian mean estimation, we are interested in recovering the mean of the dominant mixture component.
The bad samples are generated from another Gaussian distribution, and the distance of the two centers is set as 1 for convenience. We test the performance when τ (cid:63) varies in the interval [0 . 5 , 0 . 8] with σ = 0 . We compare: (1) MLE :
All samples are treated as good samples to ﬁnd the best parameter; (2) Oracle : learn from the τ (cid:63) n good samples given oracle access; (3) Super-Oracle : learn from n samples given oracle access; (4) R-Init+ILFB : ILFB with random initialization; (5) M-Init+ILFB : ILFB using MLE as initialization.
According to Fig. 9 (left), ILFB performs close to the oracle, and is much better than the naive method and ILFB with random initialization. In Fig.
9 (right), we see linear convergence at the ﬁrst few rounds until reaching a static point.
[Figure]
D.2. A SYMPTOTIC PERFORMANCE
Next, we check the asymptotic behavior of ILFB for Gaussian mixture model under different noise levels.
As we stated in our theorem, ILFB will converge to the noise ball centered at the true parameter, which does not guarantee consistency.
In fact, our experiments veriﬁes that the result is tight.
Our experimental setting is as follows: we consider mixture of two Gaussian distributions, where 60% of the data comes from the interested component.
The distance between two centers is 1 .
We test the performance under different sample sizes, varies from 100 to 51200 .
The standard deviation of each result is calculated based on 100 random runs.
In Fig.
10 (middle), there exists a gap between ILFB and the oracle, i.e., the oracle goes asymptotically to the true parameter as n gets larger, while the performance of ILFB does not decrease with n after some threshold.
The same problem exists when the noise becomes larger, as shown in Fig.
10, however, for n ≤ 51200 , the results for both ILFB and the oracle are dominated by the noise.
In the small noise regime, both ILFB and the oracle perform well.
We also want to point out that this pheonema of non-exact recovery when the noise is large is not a problem introduced by our algorithm.
In fact, for a series of work on learning Gaussian mixture models (estimating the means given identity covariance matrix), a separation between every pair of mixture components is required in order to learn in polynomial time (notice that this polynomial time is usually much larger compared to our algorithm which is near-linear).
For Fig.
10 (middle and right), this separation requirement is clearly not satisﬁed.
[Figure]
D. I NITIALIZATION FOR G AUSSIAN MIXTURE MODEL
We present a performance guarantee of the simple initialization step, which takes the average over all the samples.
More speciﬁcally, we show that the initialization will be closer to the dominant component under mild assumptions.
Theorem 7 ( initialization).
Given a dataset D generated following (2), for a certain j , deﬁne d jm :=
min l ∈ [ m ] \{ j } (cid:107) θ ( j ) − θ ( l ) (cid:107) 2 .
Consider taking the overall mean as the initialization θ 0 , then, if (cid:13)(cid:13)(cid:13)(cid:80) l ∈ [ m ] \{ j } ( θ ( j ) − θ ( l ) ) | S l | (cid:13)(cid:13)(cid:13) 2 ≤ αnd jm , and n ≥ cdσ 2 ( 919 − α ) 2 ( d jm ) 2 , we have (cid:107) θ 0 − θ ( j ) (cid:107) 2 ≤ 0 .
9 min l ∈ [ m ] \{ j } (cid:13)(cid:13) θ 0 − θ ( l ) (cid:13)(cid:13) 2 with high probability.
Proof.
Assume (cid:13)(cid:13)(cid:13)(cid:80)
j ∈ [ m ] ( θ (cid:63) − θ ( j ) )
| S j bad | (cid:13)(cid:13)(cid:13) 2 ≤ αn min j ∈ [ m ] (cid:13)(cid:13) θ (cid:63) − θ ( j ) (cid:13)(cid:13) 2 .
Consider the initialization θ 0 = 1 n n (cid:88) i =
x i .
We have with high probability, (cid:107) θ 0 − θ (cid:63) (cid:107) 2 = 1 n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i
= x i − θ (cid:63) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88) i ∈
S good x
i − θ (cid:63) + (cid:88) i ∈
S bad x
i − [Table] if n ≥ [Table]
This will imply
[Table]
(cid:4)
E M ORE EXPERIMENTAL RESULTS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Experimental settings: In this section, we present additional experimental results, in order to verify the performance of ILFB under different parameter settings, and compare with other algorithms.
More speciﬁcally, we present the results using the following methods/algorithms:
• Baseline : naive trainig using all the samples; • ILFB : our proposed iterative learning algorithm with 5 iterations, using a mis-speciﬁed τ which is 5% less than the true value; • Centroid : using the centroid of the input data to ﬁlter out outliers.
For classiﬁcation task, we calculate the centroids for the samples with the same label/class and ﬁlter each class separately; • 1-Step : ILFB algorithm with a single iteration; • ∆ τ = τ (cid:63) − τ ∈ { 10% , 15% , 20% } : ILFB under different mis-speciﬁed τ value, under the following three problem settings: • Classiﬁcation with systematic error ; • MNIST generation with Fashion-MNIST images ; • Classiﬁcation with random label error .
For classiﬁcation tasks (Table 4 and Table 6), we show the best accuracy on the validation set.
For the generation task (Table 5), we present the ratio of true MNIST samples selected by each method.
For the baseline method, since the DC-GAN is trained using all samples, the reported value is exactly the τ (cid:63) .
Results: Table 4 shows the performance under systematic error, for τ (cid:63) varies from 60% to 90% .
ILFB not only performs better than the baseline, but also outperform the centroid method by a large margin, especially when the dataset is noisy ( τ (cid:63) is small).
By comparing ILFB with its 1 -step counterpart, we see the beneﬁt brought by doing the iterative updating is also signiﬁcant.
Tableshows the performance of generation quality under different noise levels.
We observe that centroid method does not work, which may due to the fact that all MNIST and Fashion-MNIST images are hard to be distinguished as two clusters in the pixel space.
Notice that there are in fact 20 clusters (from MNIST, and 10 from Fashion-MNIST), and we are interested in 10 of them.
ILFB works well since it automatically learns a clustering rule when generating on the noisy dataset.
For example, for τ (cid:63) = 80% , even with a mis-speciﬁed τ = 60% , ILFB is capable of ignoring almost all bad samples.
Again, we also observe signiﬁcant improvement of ILFB over its 1 -step counterpart.
In Table 6, we show the performance of ILFB in the random error setting.
Again, ILFB performs much better in the extremely noisy setting.
We also have results showing that ILFB works well for generation when the corrupted samples are pure Gaussian noise.
However, we do not think it is a practical assumption, and the result is not presented here.