L EARNING TO A BSTAIN IN THE P RESENCE OF U NINFORMATIVE D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning and decision making in domains with naturally high noise-to-signal ratios – such as Finance or Public Health – can be challenging and yet extremely important.
In this paper, we study a problem of learning on datasets in which a signiﬁcant proportion of samples does not contain useful information.
To analyze this setting, we introduce a noisy generative process with a clear distinction between uninformative/not learnable/purely random data and a structured/informative component.
This dichotomy is present both during the training and in the inference phase.
We propose a novel approach to learn under these conditions via a loss inspired by the selective learning theory.
By minimizing the loss, our method is guaranteed to make a near-optimal decision by simultaneously distinguishing structured data from the non-learnable and making predictions, even in a highly imbalanced setting.
We build upon the strength of our theoretical guarantees by describing an iterative algorithm, which jointly optimizes both a predictor and a selector, and evaluate its empirical performance under a variety of conditions.
I NTRODUCTION
Despite the success of machine learning in computer vision (Deng et al., 2009; Krizhevsky et al., 2009;
He et al., 2016a; Huang et al., 2017) and natural language processing (Vaswani et al., 2017;
Devlin et al., 2018), the power of ML is yet to make signiﬁcant impact in other areas.
One major challenge is the inherently high noise-to-signal ratio in certain domains.
For example, in Finance, while stock prices generally reﬂect the information about ﬁnancial health of their companies, over the short term, their ﬂuctuations most closely resemble random walks - which are naturally unpredictable - and are usually modeled as such (Tsay, 2005).
In biomedical research, the underlying phenomena are often highly complex and are affected by unobservable factors.
The outcome may appear highly random to the measurements (gene expression, medical histories), if the true causing factor is not included or is overwhelmed by others.
We are interested in dealing with datasets that may contain large fraction of noisy/uninformative/not learnable data in both training and testing stages.
Direct application of standard supervised learning methods to such datasets is both challenging and unwarranted.
At the training stage, the uninformative data can signiﬁcantly bias the model or even completely overwhelm the true signal (Nettleton et al., 2010).
Therefore, naïve forecasts in majority-uninformative datasets are doomed to be unreliable.
Compared to other learning methods, deep neural networks are even more affected by the presence of noise, due to their strong memorization power (Zhang et al., 2017): they are likely to overﬁt the noise and make overly conﬁdent predictions where no real structure exists.
In this paper, we propose a novel method for learning on datasets where a signiﬁcant portion of content is pure noise.
Instead of forcing the classiﬁer to make predictions for every sample, we learn to decide whether a datapoint is informative or not.
If successful, the method abstains from making decisions where no structure exists, and predominantly learns from the remaining predictable data.
Our idea is inspired by the classic selective prediction problem (Chow, 1957), in which one learns to select a subset of data and only predict on that subset.
However, the goal of selective prediction is very different from ours.
A selective prediction method considers all data relevant.
It pursues a balance between coverage (i.e. proportion of the data selected) and conditional accuracy on the selected data.
In our problem, we assume that uninformative data is an integral part of the data generative process.
No learning method, no matter how powerful, can be successful on such data.
Our goal is to identify these uninformative samples as well as possible, and at the same time, to train a classiﬁer by minimizing conditional risk on the remaining informative data.
Our method learns both a predictor , f , that classiﬁes samples, and a selector , g , that selects learnable data for the predictor and rejects/abstains from the uninformative data.
We are using g to approximate the ground truth indicator function of structured/informative data, g ∗ .
We assume that g ∗ exists as a part of the data generation process, but it is never revealed to us, even during training.
Instead of direct supervision, we therefore must rely on the predictor’s mistakes to train the selector.
To achieve this goal, we propose a novel selector loss enforcing that (1) the selected data best ﬁts the predictor, and (2) the portion of the data where we abstain from forecasting, does not contain many correct predictions.
This loss function is quite different from the loss in classic selective prediction, which penalizes all unselected data equally.
A major contribution of this paper is the derivation of theoretical guarantees for the empirical minimizer of our loss.
We analyze the proposed selector loss function and provide sample complexity for learning a nearly optimal selector.
We show that optimizing such loss can recover nearly all the structured/informative data in a PAC fashion (Valiant, 1984; Kearns et al., 1994; Valiant, 2013), i.e. one can approximate the ground truth selector function g ∗ well with high probability, given sufﬁcient samples.
What may be surprising is that this guarantee holds even in a challenging setting where the uninformative data represents the majority of the training set.
This theoretical guarantee lets us expand to a more challenging and realistic setting.
When the sample size is limited and the initially learned predictor is not sufﬁciently close to the ground truth, we extend our method to an iterative algorithm, in which we progressively optimize both the predictor and the selector.
The selector is improved by optimizing our novel selector loss.
Meanwhile, the predictor is improved by optimizing the empirical risk, reweighted based on the selector’s output; uninformative or nearly-uninformative samples identiﬁed by the selector will be down-weighed.
Experiments on real-world datasets demonstrate superiority of our method to existing baselines.
Note that in this paper, we assume that each sample is either informative or not.
Extending our method to a more general setting with continuous transitions between informative and uninformative is non-trivial and is left as future work.
1. R ELATED W ORK
Learning with untrusted data aims to recover the ground truth model from a partially corrupted dataset.
Different noise models for untrusted data have been studied, including random label noise (Bylander, 1994; Natarajan et al., 2013; Han et al., 2018;
Yu et al., 2019; Zheng et al., 2020; Zhang et al., 2020), bounded label noise (Massart & Nédélec, 2006; Awasthi et al., 2015; Diakonikolas et al., 2019; 2020) and adversarial noise (Kearns & Li, 1993; Kearns et al., 1994; Kalai et al., 2008;
Klivans et al., 2009; Awasthi et al., 2017).
In the most pessimistic setting, if the majority data is corrupted by arbitrary adversarial noise, even mean estimation may be impossible.
This is known as List-Decodable problem (Balcan et al., 2008; Charikar et al., 2017; Diakonikolas et al., 2018), where the best one can do when the proportion of trusted data α is less than 0.5, is to return 1 α many hypotheses of the mean, knowing one of them is promising.
We, on the other hand, aim to produce a single accurate model even in a setting where the majority of the data is uninformative (the noise, of course, must be of a certain type, see next section).
While above works assume the presence of noisy data only in the training stage, we study the case where noise is an integral part of the generative process and thus will appear during inference as well, where it must be detected and discarded once more.
Selective learning is an active research area.
It extends the classic selective prediction problem and studies how to select a subset of data for different learning tasks.
We can summarize existing methods into 4 categories: Monte Carlo sampling based methods (Gal & Ghahramani, 2016; Kendall & Gal, 2017; Pearce et al., 2020), margin based methods (Fumera & Roli, 2002; Bartlett & Wegkamp, 2008; Grandvalet et al., 2008; Wegkamp et al., 2011; Zhang et al., 2018), conﬁdence based methods (Wiener & El-Yaniv, 2011; Geifman & El-Yaniv, 2017; Jiang et al., 2018) and customized selective loss (Cortes et al., 2016; Geifman & El-Yaniv, 2019; Liu et al., 2019).
Notably, several works propose customized losses, and incorporate them into neural networks.
In (Geifman & El-Yaniv, 2019), the network maintains an extra output neuron to indicate rejection of datapoints.
Liu et al.
(2019) use
Gambler loss where a cost term is associated with each output neuron and a doubling-rate-like loss function is used to balance rejections and predictions.
Cortes et al.
(2016) perform data selection with an extra model and introduce a selective loss that helps maximize the coverage ratio, thus trading off a small fraction of data for a better precision.
Existing works on selective prediction are all motivated from the coverage perspective - i.e. one wants to make safe prediction to achieve higher precision while maintain a reasonable recall (El-Yaniv et al., 2010).
Whereas our paper is the ﬁrst to investigate the case where some (or even majority) of the data is uninformative, and thus must be discarded at prediction time.
Unlike with the selective prediction, there is a latent ground truth indicator function of whether a datapoint should be selected or not.
Our method is guaranteed to identify those uninformative samples.
P ROBLEM F ORMULATION
In this section, we describe the inherently-noisy data generation process that we aim to study.
The model has three important features: 1) the uninformative portion of the data has labels generated by coin ﬂipping; 2) the informative datapoints are labeled with a latent ground truth function; and 3) the uninformative data has a distinguishable support from the informative data.
Formally:
Deﬁnition 1 (Noisy Generative Process) .
We deﬁne Noisy Generative Process by the following notation x ∼ D α where
[Equation]
prob
[Equation]
(Uninformative/Noisy Data)
[Equation]
with prob
[Equation]
(Informative/Structured Data
[Equation]
Let Ω D ⊆ R
d be the support of D α .
Suppose { Ω U , Ω I } is a partition of Ω D , the ground truth labeling function f ∗ :
X → { +1 , − 1 } is in hypothesis class F and data is sampled according to:
[Equation]
Note that the f ∗ is deﬁned on the whole domain, although only the part within Ω I is relevant.
α represents the fraction of informative or structured data in the population.
For the rest of the paper, we abuse the notation and use ( x, y ) ∼ D
α to refer to samples generated from this process.
Next deﬁnition describes a separability condition between informative and uninformative data.
It allows to distinguish noise from structure and enables us to approximate the ground truth selector via empirical minimization.
Deﬁnition 2 ( H -Separable) .
Given compact set Ω and its partition { Ω U , Ω I } , { Ω U , Ω I } satisﬁes
H -Separable condition if there exists g ∗ ∈ H satisfying g ∗ ( x ) :
X → { +1 , − 1 } :
[Equation]
One can view g ∗ ( · ) in deﬁnition 2 as the target selector we wish to recover.
Having introduced the data generation process and the separability condition, we now describe our main assumption for the remainder of the paper.
Assumption 1.
Data S n = { x i , y i } Ni =1 is i.i.d generated according to the Noisy Generative Process (Deﬁnition 1), with f ∗ ∈ F , support Ω D α = Ω U ˙
∪ Ω I where Ω U , Ω
I are H -Separable.
Throughout this paper, we are interested in the following learning task:
Problem 1 (Abstain from Uninformative Data) .
Under Assumption 1 with enough i.i.d observations from D α : 1) given hypothesis class F , we aim to learn the underlying ground truth classiﬁer f ∗ ( x ) ; and 2) given the hypothesis class H , we aim to learn the selector g ∗ ( x ) .
Evaluation metrics.
We deﬁne metrics to evaluate the quality of both the prediction and the selection.
For the prediction, we borrow the selective risk deﬁnition from selective learning (El-Yaniv et al., 2010; Geifman & El-Yaniv, 2019; 2017) using our own notation.
Deﬁnition 3 (Selective Risk) .
Given a predictor f ∈ F and a selector g ∈ H , we deﬁne the selective risk as: CR ( f, g ) =
E ( x,y )
∼D α [ 1 { f ( x ) (cid:54) =
y }| g ( x ) ≥ 0] and its empirical version:
CR S ( f, g ) = (cid:80)
[Table]
Selective risk measures the average risk conditioned on instances that are picked by the selector g ( x ) .
Note here when combined with the ground truth selector g ∗ , the selective risk of the ground truth predictor f ∗ goes to zero.
Without a selector, however, f ∗ has classical classiﬁcation risk (which will be formally deﬁned in Deﬁnition 5 later) of more than 12 (1 − α ) .
The metric used to evaluate the quality of a learned selector g is its false positive/negative rate.
When g ( x ) ≥ 0 and g ∗ ( x ) < 0 , the selector g accepts an uninformative datapoint and thus commits a false positive error .
When g ( x ) < 0 and g ∗ ( x ) ≥ 0 , g rejects/abstains from an informative datapoint resulting in a false negative error .
Deﬁnition 4 (Evaluation Metric for the Selector) .
Given distribution D α as deﬁned in Defnitionand g ∗ as target selector, we denote the false positive and false negative of a selector g as
False Positive
[Equation]
P
[Equation] [Equation]
False Negative
[Equation]
O UR M ETHOD
In this section, we present our approach for learning and abstaining in the presence of uninformative data (Problem 1).
The main challenge is that the latent informative/uninformative status of a datapoint is unknown.
Our main idea is to introduce a novel selector loss function that trains a selector based on the performance of the best predictor (Section 3.1).
In Section 3.2, we present our main theoretical result.
We show that by jointly ﬁnding a predictor minimizing the classiﬁcation risk and ﬁnding a selector minimizing the proposed selector loss, we can solve Problem 1 with controlled selective risk and selector error rates.
During the analysis, we assume an oracle is given for the empirical minimizer.
Approximating such empirical minimizer is a different topic beyond the scope of this paper (Bartlett et al., 2006; Yuan & Wegkamp, 2010).
Inspired by the theoretical results, in Section 3.3, we propose a heuristic algorithm that iteratively optimizes the predictor and the selector.
Throughout the analysis, we focus on the 0-1 classiﬁcation loss to optimize the predictor.
This follows the standard for analyzing generalization performance and sample complexity.
Formally:
Deﬁnition 5 (Classiﬁcation Risk) .
We denote the classiﬁcation risk of a model f as
[Equation]
and the empirical classiﬁcation risk: R S n ( f ) = 1 n (cid:80) ni
=1 1 { f ( x i ) (cid:54) =
y i } .
We deﬁne f ∗ S n to be the empirical minimizer of R S n ( f ) : f ∗ S n = arg min f ∈F R S n ( f ) .
3. S ELECTOR L OSS
To learn a selector without direct supervision, we have to leverage the performance of a given predictor, f .
We propose the selector loss , which reweigh the 0 - 1 loss based on the prediction of f .
This loss penalizes when (1) the predictor makes a correct prediction on a datapoint that the selector considers uninformative and abstains from, or (2) the predictor makes an incorrect prediction on a datapoint that the selector considers informative.
Deﬁnition 6 (Selector Loss) .
Given f ∈ F and its selector g ∈ H , we deﬁne the following weighted 0 − 1 type risk w.r.t g ( · ) as selector risk:
[Equation]
and its empirical version:
[Equation]
Intuitively speaking, the loss will drive the selector to partition the domain into informative and uninformative regions.
Within the informative region, the predictor is supposed to ﬁt the data well, and should be more accurate.
Meanwhile, within the uninformative region, the label is random and the predictor is supposed to be more prone to error.
Another view of the loss is that we are learning to ﬁt the selector g ( x ) to a pseudo-informative label given by the predictor, 1 { f ( x )
= y } .
Since such label is f -dependent, the quality of f is crucial for successfully learning g .
In the theoretical analysis, we leverage the fact that predictor f is close enough to the ground truth predictor f ∗ .
Note that there are two types of errors penalized in the selector loss: an incorrect prediction on an informative datapoint, ( f ( x ) (cid:54) =
y ) ∧ ( g ( x ) > 0) , and a correct prediction on an uninformative datapoint, ( f ( x ) =
y ) ∧ ( g ( x ) ≤ 0) .
We use β I = θα and β U = 1 − θ 1 − α to weigh these two types of error in the loss.
Both β I and β U are controlled by a parameter θ and the latent informative data proportion α .
We show that for a wide range of θ , the accuracy of the selector is guaranteed, as long as α is bounded away from 1 , i.e., when abstention is necessary.
In practice, we can adjust β
I
and
β U freely without knowing exact value of α , as empirical performance remains stable with regard to these choices.
Learning a selector with the novel loss.
In an ideal setting, to learn a selector, we ﬁrst estimate the predictor by minimizing the classiﬁcation risk, f ∗ S n = arg min f ∈F R S
n ( f ) .
Next, we estimate the selector by minimizing the selector loss, conditioned on the estimated predictor, f ∗ S n , g ∗ S n = arg min g ∈H W S n ( g, f ∗ S n , θ ) .
In Figure 1, we show an example of using the empirical minimization strategy with logistic regression and with 0 - 1 loss replaced by cross-entropy loss.
In this case, the losses are all convex and the empirical minimizers f ∗ S n and g ∗ S n can be computed exactly.
Our analysis in Section 3.2 will show that the estimated predictor f ∗ S n and selector g ∗ S n has bounded selective risk (as in Deﬁnition 3), as well as bounded false positive and false negative selector errors (as in Deﬁnition 4).
In practice, however, empirical minimization is not always possible, as optimization for complex models (e.g., DNNs) and non-convex losses remains open.
We therefore propose a heuristic algorithm in the spirit of our theoretical results - it jointly learns f and g by minimizing the selector loss and a reweighed classiﬁcation risk iteratively (see Section 3.3).
[Figure]
3. T HEORETICAL A NALYSIS
In this section we report our theoretical results.
The main result can be summarized in the following (informal) statement.
Main Result (Informal) With sufﬁcient data, the estimated predictor f ∗ S n = arg min f ∈F R S n ( f ) and selector g ∗ S n = arg min g ∈H W S n ( g, f ∗ S n , θ ) are sufﬁciently close to the targets f ∗ and g ∗ with high probability.
The proof requires the classic growth function deﬁnition (Vapnik & Chervonenkis, 2015).
Deﬁnition 7 (Growth Function) .
Let H be the hypothesis class of function f and F x 1 ,...,x n = { f ( x 1 ) , ..., f ( x n ) : f ∈ F} ⊆ { +1 , − 1 } n .
The growth function is deﬁned to be the maximum number of ways in which n points can be classiﬁed by the function class: G F ( n ) = sup x 1 ,...,x n |F x 1 ,...,x n | .
Roadmap of the proof : Theorem 1 is a standard PAC Learning analysis where we show f ∗ S n is sufﬁciently close to the ground truth function f ∗ .
Note that our analysis does not imply efﬁcient PAC learning since optimizing binary loss is NP-hard.
In Theorem 2, we bound the statistical error of the selector loss.
In Theorem 3, we leverage the fact that informative and uninformative portions of the data are distinguishable, as well as the fact that f ∗ S n has low risk on informative data.
This lets us bound the selector loss of g ∗ S n given by W ( g ∗ S n , f ∗ S n , θ ) .
By carefully balancing the weight parameter θ of selector loss, we derive the false positive and false negative guarantees in Theorem 3.
Theoremis a formal version of the main result, which applies Sauer’s Lemma (Sauer, 1972) to bound the growth function of hypothesis classes in Theorem 3.
The toolkit we use is driven by the concept of a
VC-dimension in (Vapnik & Chervonenkis, 2015; Blum et al., 2016; Mohri et al., 2018).
We also provide a lower bound construction in the Appendix to show the (near) tightness of our analysis (up to a logarithmic factor).
We present all the results below, detailed proofs are in the Appendix.
We ﬁrst prove a theorem about sample complexity of the learning hypothesis optimized over a given dataset, referred to as f ∗ S n .
We show that it approximates the ground truth function
f ∗ well.
Theorem 1.
Under Assumption 1, given hypothesis class F , a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d from our Noisy Generative Process and f ∗ S n = arg min f ∈F R S ( f ) , suppose n ≥ 32 [ log(4 |G F (2 n ) | )
+log( 1 δ ) ] (cid:15) 2 α, then with probability at least 1 − 2 δ,
R ( f ∗ S n ) ≤ 12 (1 − α )
+ 2 (cid:15)α.
Furthermore, P x ∼D I [ f ∗ ( x ) (cid:54) =
f ∗ S n ( x )] ≤ 2 (cid:15).
Next theorem states the sample requirements in order to achieve a small gap between the empirical selector loss and the true selector loss.
The proof takes a union bound over all possible pairs of hypothesis ( g, f
) ∈ H × F , which explains the term |G H (2 n ) ||G F (2 n ) | in the bound for n , the minimum number of sample points.
Theorem 2.
Given hypothesis class H , F , a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process and f ∗ S n = arg min f ∈F R S ( f ) , g S n ∗ = arg min g ∈H W S n ( g, f ∗ S n , θ ) , if θ ∈ ( α, 1) , α ∈ (0 , 1) and n ≥ 32 θ 2 [ log (4 |G H (2 n ) ||G F (2 n ) | )
+log( 1 δ ) ] (cid:15) 2 α, with probability at least 1 − δ , | W
[Equation]
g
[Equation]
S n , f
[Equation]
S n ,
[Equation]
n (
[Equation] [Equation]
(cid:15)α.
Theorem 3 combines the results in Theorems 1 and 2.
It shows that, by carefully choosing weight parameter θ , minimizing the selector loss with sufﬁciently large amount of data ensures a low false positive/negative rate of the selector with high probability.
In particular, the analysis points to a trade off between false positive and negative rates faced by the selector in terms of θ .
We characterize the relationship between θ and α using variable c = θ 2 α in the theorem and prove that for a wide range of c the false negative rate (FNR) and false positive rate (FPR) are controlled.
Intuitively, a larger value of θ in W S n ( g, f ∗ S n , θ ) implies a higher penalty on the false positive error.
Theorem 3.
Under Assumption 1, given a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process and f ∗ S n = arg min f ∈F R S ( f ) , g S n ∗ = arg min g ∈H W S n ( g, f ∗ S n , θ ) , if θ ∈ ( α, 1) , α ∈ (0 , 1) and n ≥ 32 θ 2 [ log (4 |G H (2 n ) ||G F (2 n ) | )
+log( 5 δ ) ] (cid:15) 2 α, if θ = 2 cα with 12 < c < 1+ α 4 α , we have with probability at least 1 − δ
[Equation]
Next theorem is a formal statement of the main result.
It is worth mentioning that the theorem assumes that α is bounded away from 1 , due to the fact that when almost all data is informative, there is no need for abstention.
Theorem 4.
Under Assumption 1, given set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process, hypothesis class H with VC-dimension d vc ( H ) , F with VCdimension d vc ( F ) and f ∗ S n = arg min f ∈F R S ( f ) , g S n ∗ =
arg min g ∈H W S n (cid:16) g, f ∗ S n , 73 α
+ α 2 72 (cid:17) , if α ∈ (0 , 0 . 9) and n ≥
[ ( d vc ( H )+ d vc ( F
[Equation]
log
[Equation]
+log
[Equation] [Equation]
) ] (cid:15)
[Equation]
We have with probability at least 1 − δ :
I
[Equation]
False Positive
[Equation]
II
[Equation]
Remark 1.
We provide a lower bound construction in the appendix to show that our bound in
Theorem 4 is tight up to a logarithmic factor.
Note that our lower bound is for empirical minimizers of R ( f ) and W ( g, f, θ ) , therefore the tightness result is not an information theoretic bound for this family of problems.
It remains an open question about whether or not strategies other than the empirical minimization, as in (Hanneke, 2016; Simon, 2015; El-Yaniv et al., 2010), can improve the sample complexity to a rate of O (cid:0) 1 αε (cid:1) .
Note that the informative portion of the data (drawn with probability α ) by itself only requires O ( 1 ε ) samples for an (cid:15) -excessive statistical error.
It turns out that our analysis for empirical minimizer is nearly tight, and that the number of samples O (cid:0) 1 α 2 ε 2 (cid:1) is indeed necessary because the presence of noisy data makes our problem harder.
Remark 2.
Theorem 4 says that by minimizing the empirical version of the loss from Deﬁnitionsand 6, one can achieve the goal of simultaneously selecting and learning with noisy data - i.e. the classiﬁcation model f ∗ S n has sufﬁciently low selective risk, and the selector g ∗ S n can distinguish informative data from non-informative.
The analysis of the selector loss (Theorem 3) relies on the quality of the classiﬁer f ∗ S n .
But since we know that g ∗ S n is able to abstain from uninformative data, we can retrain f ∗ S n on the informative portion only, therefore guaranteeing its accuracy.
Such circular logic naturally leads to a practical iterative algorithm that we present in the next section.
3. A H EURISTIC A LGORITHM IN P RACTICE
In Section 3.2, we analyzed an empirical minimizer of a conceptual 0 - 1 loss for the classiﬁer f and the selector g .
Binary loss, however, is impractical from the computational standpoint.
In practice, we use cross-entropy loss instead and require that both f and g have continuous-valued output, ranging between 0 and 1.
We also relax the requirement for empirical minimization oracles, allowing the practical algorithm to jointly optimize the predictor and the selector in an iterative manner.
At each iteration, we update the predictor using the informative data selected by the selector, and then update the selector based on the predictor’s output.
See Algorithm 1 for the pseudo-code.
A pictorial example of Algorithm 1’s performance can be found in Figure 2 in the Appendix.
During the joint optimization process, the predictor is counting on the selector to show it only the informative data.
However, the initial selector is not trustworthy.
To update the predictor f , we turn to a so-called soft abstention scheme: use a weight vector γ that progressively down-weighs samples abstained by g , in the spirit of multiplicative weights algorithms (Cesa-Bianchi & Lugosi, 2006; Arora et al., 2012).
Speciﬁcally, we increase the weight of i -th sample γ
i if the selector accepts x i : γ i = γ
i (1 + η · 1 { g ( x i ) > 0 . 5 } ) and then normalize so that (cid:80) ni
=1 γ i = 1 .
We call this a soft abstention approach because the algorithm decreases the weight of uninformative data gradually.
In the heuristic algorithm, we use cross-entropy loss to ﬁt the selector to pseudo-informative labels output by the predictor.
We also use a single hyper-parameter β
= β
I /β U
= θ (1 − α )
α (1 − θ ) to control the ratio of the two types of errors.
As suggested by Theorem 3, small selector FPR and small selector
FNR can be achieved by a β ∈ (cid:0) 1 , 1 + 1 α (cid:1) .
For a scenario when abstention is necessary, i.e. when the noisy data ratio is in a reasonable range, we can choose our hyper-parameter from a wide interval.
Require: Data set S n = { ( x 1 , y 1 ) , ..., ( x n , y n )) } , weight parameter: β , random initial f 0 and g 0 , initial sample weights γ
i = 1 n , ∀ i ∈ [ n ] , meta learning rate η , number of iterations T 1: for t ← 1 , · · · , T do 2:
Optimize loss to update predictor f t : (cid:80) ni
=1 γ ti { y i log( f ( x i ))
+ (1 − y i ) log(
− f ( x i )) } 3:
Approximate the ‘pseudo-informative label’ :
z ti = 1 { 1 { f t ( x i ) > 0 .
= y i } 4:
Optimize loss to update selector g t : (cid:80) ni
=1 { z ti log( g ( x i ))
+ β (1 − z ti ) log(1 − g ( x i )) } 5:
Update sample weights using g t : γ t +1 i = γ ti (1+ η 1 { g t ( x i ) > 0 . 5 } ) (cid:80) nj =1 γ tj (1+ η 1 { g t ( x j ) > 0 . 5 } ) .
6: end for 7: Output f T , g T
E XPERIMENTS
Datasets.
We test the efﬁcacy of our heuristic algorithm (Algorithm 1), via semi-synthetic experiments.
We synthesize the noisy data by mixing datasets from two different domains and uniformly shufﬂing all labels in one of them.
Our ﬁrst synthetic noisy dataset is composed of MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).
We uniformly shufﬂe the labels of MNIST images and keep original labels of Fashion-MNIST.
As a result, images from MNIST are uninformative while images from Fashion-MNIST are considered informative.
The fact that one dataset has hand-written digits and the other one contains pictures of clothes ensures distinct supports.
We mix these two datasets in different proportions to proxy for our Noisy Generative Process.
We also perform experiments on another synthetic dataset, where noisy and informative portions come from similar domains.
To achieve this, we use SVHN (Netzer et al., 2011).
We uniformly at random ﬂip labels of classes 5-9 into classes 0-4 and mix them with intact data from classes 0-4.
In this case, we consider the data from classes 5-9 uninformative.
In this mixed dataset, informative and uninformative data have similar domains, which makes it more challenging to correctly identify informative data (and harder to argue separate support).
Baselines.
We compare our method to two of the most recently proposed selective learning algorithms.
(1) SelectiveNet (Geifman & El-Yaniv, 2019), which integrates an extra neuron as a data selector in the output layer and also introduces a loss term to control the coverage ratio; (2) DeepGambler (Liu et al., 2019), which also maintains an extra neuron for abstention and uses a doubling-rate-like loss term (i.e., gambler loss) to train the model.
(3) We also create a third baseline that selects data using model prediction conﬁdence, which we refer to as Conﬁdence .
The hypothesis is that informative data should have higher prediction conﬁdence compared to uninformative data.
Evaluation Metrics.
We use three criteria to jointly evaluate a selective learning outcome.
Selective risk (SR).
Selective risk is the empirical risk measured over the datapoints selected by the algorithm.
This is a metric that is also adopted in (Geifman & El-Yaniv, 2019; Liu et al., 2019).
Precision
Experiment setting.
We assume that the ratio of informative data, α , is unknown to all methods.
This is necessary in practice; the ratio and strength of noise are not known in most real world scenarios.
For SelectiveNet and DeepGambler, such ratio is a required input.
To run these baselines, we ﬁrst run the original backbone for 60 epochs, and then estimate α using backbone’s training accuracy.
Assume that the backbone ﬁts all of the informative data perfectly and also makes some correct guesses on noisy data with probability 1 num of classes , then the frequency estimation of α is
, which is the value we give to baselines.
For more details about the experimental design, please refer to the Appendix (section D).
Results and discussion.
Table 1 presents the result where we mix the entire uninformative dataset (shufﬂed MNIST) with different proportions of the informative dataset (Fashion-MNIST).
Our algorithm outperforms all the baselines in this setting.
If we use partial dataset by decreasing the number of noisy datapoints and repeating previous experiments with the same uninformative-toinformative data ratios, our algorithm wins by an even larger margin (Table 2).
We can see that both SelectiveNet and DeepGambler fall apart in the setting where informative data are sparse.
Neither of them achieve a risk below 40% when we mix in only 25 % of informative data.
The reason is that both of these two methods are only designed to achieve predetermined coverage and cannot dynamically adapt to settings with unknown informative-to-uninformative data ratio.
Our algorithm thus gains advantage by learning the underlying uninformative data labeling function in a data-driven manner.
Our algorithm also outperforms all the baselines on SVHN dataset where noisy data comes from a similar domain to the informative data.
Each experiment is repeated 5 times.
Mean and standard deviation are presented in Table 3 and Table 4.
We provide more experimental results in the Appendix (section D).
We test the performance of each method in the case where α is given and in the case where datasets contain minority noisy data.
[Table] [Table]
[Table] [Table]
ONCLUSION
In this work, we propose a natural noisy generative process to model a problem where a signiﬁcant proportion of datapoints are purely random.
We solve it by learning both a predictor for classiﬁcation and a selector to abstain from the uninformative data.
We propose a novel selector loss to learn selector with theoretical guarantees.
Based on the selector loss, we design a heuristic algorithm that jointly learns the predictor and selector.
Our empirical study shows promising results of our method.
R EPRODUCIBILITY S TATEMENT
We describe a synthetic data generation procedure, evaluation metrics and experiment setting in section 4.
For the convenience of the reader to reproduce the experiment, we also summarize the setting and give implementation details in section D.3.
The source code as well as parameters to reproduce the experimental results will be made available together with the publication of the paper.
E THICS S TATEMENT
This paper focuses on a theoretical discussion about learning from data that contains different portion of non-informative samples.
Our experiments only use publicly available datasets.
Our discussion, analysis, or data shouldn’t raise any ethics-related issues.
The learning method proposed in this paper, however, can be potentially used in applications with fairness and privacy concerns.
It our future efforts in this area, we aim to address and resolve possible negative impact
A I LLUSTRATIVE E XAMPLE FOR A LGORITHM
[Figure]
B T HEORETICAL A NALYSIS
In this section, we show auxiliary results culminating with the proof of the main result in Theorem 4.
In this section, we will use the following well-known result repeatedly.
Lemma 1 (Hoeffding’s Inequality) .
Let Z 1 , ..., Z n be independent bounded random variables with
Z i ∈ [ a, b ] for all i , where −∞ < a < b < ∞ .
Then for all t > 0 :
[Equation]
The next lemma provides an upper bound on the empirical risk R S ( f ∗ ) , if enough samples are available.
Lemma 2.
Consider a set of samples S
= { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy
Generative Process and f ∗ in the hypothesis class F satisfying f ( x ) ∈ {− 1 , +1 } .
If:
[Equation]
Then we have with probability at least 1 − δ :
[Equation]
Proof : By deﬁnition:
[Equation]
(11) and the empirical risk
[Equation]
Note that 1 { f ( x ) (cid:54) =
y } is bounded in the interval [0 , 1] and given f ∗ ∈ F , 1 { f ( x i ) (cid:54) =
y i } , i ∈ [ n ] form a set of n independent random variables.
By setting b − a = 1 , t = α(cid:15) in Equation 9, the choice of n ensures that − 2 nt 2 ( b − a ) 2 ≤ 6 log( δ ) .
Thus
[Equation]
where we have
[Equation]
This way we have:
[Equation]
which implies that Equation 10 holds with probability at least 1 − δ .
The proof of Lemma 2 relies on the independence between 1 f ( x i ) (cid:54) =
y i across different pairs ( x i , y i )
∈ S n .
However, there is no guarantee that such independence holds for the empirical minimizer f ∗ S n = arg min f ∈F R S ( f ) . .
In order to prove Theorem 1, we use the growth function in Deﬁnition 7 for sets of size n and the following symmetrization lemma.
Here we show its proof for completeness; it can be found in different sources (Mohri et al., 2018; Blum et al., 2016).
Lemma 3.
Suppose S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } are i.i.d sampled , L ( f, x, y ) ∈ [0 , b ] and L S n =n (cid:80) ni
=1 L ( f, x i , y i ) .
If
[Equation]
then we have:
[Equation]
Proof : Since | L S n ( f ) − L ( f )
| ≥ t and | L S (cid:48) n ( f ) − L ( f ) | ≤ t 2 implies | L S n − L S (cid:48)
n | ≥ t 2 thus
[Equation] [Equation]
Taking expectation w.r.t S n ∼ D α and S (cid:48) n ∼ D α we have
[Equation]
Next we lower bound P [sup f ∈F | L S (cid:48) n ( f ) − L ( f ) | ≥ t 2 ] .
Since L ( f, x, y ) ∈ [0 , b ] and V ar
( L ( f, x, y ))
≤ b 2 4 , we have:
[Equation]
Given the choice of n ≥ 2 b 2 , we have P [sup f ∈F | L S
[Equation]
n ( f ) − L ( f ) | ≥ t 2 ] ≥ 12 which implies
[Equation]
Theorem 1.
Under Assumption 1, given a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process and
[Equation]
suppose
[Equation]
Then with probability at least 1 − 2 δ :
[Equation]
Furthermore,
[Equation]
Proof.
: We ﬁrst bound the probability of the event that R ( f ∗ S n ) ≤ 12 − (1 − 2 (cid:15) ) α.
We start with bounding P S n ,S (cid:48) n ∼D α [ | R S n ( f ) − R ( f ) + R ( f ) − R S (cid:48) n ( f ) | ≥ t 2 ] for a ﬁxed f .
Since | R S n ( f ) − R ( f ) + R ( f ) − R S (cid:48) n ( f ) | ≥ t 2 implies | R S n ( f ) − R ( f )
| ≥ t 4 or | R ( f ) − R S (cid:48) n ( f ) | ≥
[Equation]
we have
[Equation]
Let F S n ,S (cid:48)
n = { f ( x ) : f ∈ F , x ∈ S n ∪ S (cid:48) n } ⊆ { +1 , − 1 } 2 n with Lemma 3 and Eq 16 we have:
[Equation]
By setting t = α(cid:15) and n ≥ 32(4log( G F (2 n ))
+log( 1 δ ))
α 2 (cid:15)we have with probability of at least 1 − δ :
[Equation]
Next we apply the fact that f ∗ S n = arg min f ∈F R S ( f ) .
Since R S n ( f ∗ ) ≥ R S n ( f ∗ S n ) , we have:
[Equation]
Since R S n ( f ∗ ) ≤
[Equation]
1 − α )
+ (cid:15)α with failure probability at most δ (Lemma 2), we have with probability at least 1 − 2 δ :
[Equation]
Next we prove the claim that:
[Equation]
Since R ( f ∗ S n ) ≤
[Equation]
+
[Equation] [Equation]
Next we analyze the weighted risk in Deﬁnition 6 and its empirical version.
In particular, we are interested in the case where θ > α , which implies θα ≥ 1 − θ 1 − α .
For a given pair ( x i , y i ) , we have:
[Equation]
It is useful to write down explicitly the following decomposition of W ( g, f, θ ) :
W
[Equation]
g
[Equation]
, θ
[Equation]
x ) ≤ 0] + 1 − θ 1 − α P ( x,y )
∼D α [ f ( x ) =
y | g ( x ) ≤ 0 , g ∗ ( x ) ≤ 0] + 1 − θ 1 − α P ( x,y )
∼D α [ f ( x ) =
y | g ( x ) ≤ 0 , g ∗ ( x ) > 0] P x ∼D α [ g ( x ) ≤ 0 , g ∗ ( x ) > 0]
= θ α P ( x,y )
∼D α [ f ( x ) (cid:54) =
y | g ( x ) > 0 , g ∗ ( x ) > 0]
P x ∼D α
[ g ( x ) > 0 , g ∗ ( x ) > 0]
+ θ 2 α P x ∼D α [ g ( x ) > 0 , g ∗ ( x ) ≤ 0] + 1 − θ 2(1 − α ) P x ∼D α
[ g ( x ) ≤ 0 , g ∗ ( x ) ≤ 0] + 1 − θ 1 − α P ( x,y )
∼D α [ f ( x ) =
y | g ( x ) ≤ 0 , g ∗ ( x ) > 0]
P x ∼D α [ g ( x ) ≤ 0 , g ∗ ( x ) > 0] (19)
One can see that if θ > α , then g ( x ) should reject all x ∈ Ω U since θα > 1 − θ 1 − α .
For the informative partition x ∈ Ω I , the decomposition suggests acceptance of all x such that f ( x ) =
y and rejection of f ( x ) (cid:54)
=
y .
This gives the optimal value 1 − θ 2(1 − α ) P x ∼D α [ g ∗ ( x ) ≤ 0] .
Note that the optimal value of
W ( g, f, θ ) is not necessarily achieved at ( f ∗ , g ∗ ) .
For any ﬁxed f , the optimal value of W ( g, f, θ ) is achived for g as long as g accepts all datapoints in Ω I that are correctly classiﬁed by f , and rejects all mistakes made by f in Ω I (as well as all datapoints in Ω U ).
Lemma 4.
Given a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy
Generative Process.
Suppose θ ∈ ( α, 1) , α ∈ (0 , 1) , (cid:15) ≤ 0 . 1 , and:
[Equation]
Then with probability at least 1 − δ we have:
[Equation]
Proof.
The following holds for a given ( x i , y i ) :
[Equation]
Since θ < 12 , we can apply Hoeffding’s inequality in Lemma 1 with a = 0 , b = θα .
For ﬁxed g, f ∈ F , 1 { f ( x i ) (cid:54) =
y i } , i ∈ [ n ] are set of n independent random variables.
By setting b − a = θα , t = α(cid:15) in Equation 9, the choice of n ensures that − 2 nt 2 ( b − a ) 2 ≤ 6 log( δ ) .
Therefore:
[Equation]
Recall the empirical minimizer of W S n ( g, f ∗ S n , θ ) :
[Equation]
Since f ∗ S n and g ∗ S n are data-dependent, it follows that W i ( g ∗ S n , f ∗ S n , θ ) , i ∈ [ n ] are not independent draws.
Our strategy is to use similar logic as in the proof of the Theorem 1.
We begin by ﬁnding an upper bound on the probability of W S n ( g, f, θ ) deviating from its expectation.
Then we take a union bound of all the possible pair of ( g, f ) ∈ H × F restricted to S n .
Theorem 2.
Consider a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy
Generative Process and
[Equation]
If θ ∈ ( α, 1) , α ∈ (0 , 1) and
[Equation]
then with probability at least 1 − δ :
[Equation]
Proof.
First we apply Lemma 3 with L S n ( f, g ) =
W S n ( g, f, θ ) , b = θα and t = α(cid:15) , and also using the condition n ≥ 32 θ 2 log (2 |G H (2 n ) |G F (2 n ) | )+log( 1 δ ) (cid:15)
[Equation]
≥ 2 θ 2 (cid:15)
α 4 in the lemma.
Then it follows:
[Equation]
S n
[Equation]
n
[Equation]
α [
[Equation]
] ≤ 4 |H S n
[Equation]
(cid:48)
n ||
[Equation]
n
[Equation]
(cid:48) n |
[Equation]
n ∼D α
[Equation] [Equation]
Next we bound the FPR and FNR of g ∗ S n leveraging on the fact that W S n ( g, f ∗ S n ) is sufﬁciently small.
Theorem 3.
Under Assumption 1, given a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from Noisy Generative Process and
[Equation]
if θ ∈ ( α, 1) , α ∈ (0 , 1) and
[Equation]
We have
I) with probability at least 1 − δ
[Equation]
II) if θ = 2 cα with 12 < c < 1+ α 4 α , we have with probability at least 1 − δ P
[Equation]
x )
[Equation]
x )
[Equation] [Equation]
Proof of I)
Proof.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof of II) :
Proof.
[Equation]
Next we lower bound Term 1 , 2 , 3 , 4 individually and apply upper bound on W ( g ∗ S n , f ∗ S n , θ ) in Eqfor the proof of the statement.
Term 1 :
[Equation]
Term 2 :
[Equation]
Term 3 :
[Equation]
By Theorem 1 we have with failure probability at most 2 δ :
[Equation]
P
[Equation]
∼D α
[Equation]
[Equation] [Equation]
Term 4 :
[Equation] [Equation]
Combining Equations 25, 26, 28, 29 we have:
[Equation]
and by part I ) in the theorem
[Equation]
we have with probability at least 1 − 3 δ :
[Equation]
Combined with the failure probability 2 δ in the bound of Term 3 we have the failure probability at most 5 δ :
[Equation]
By picking θ = 2 cα with 12 < c < 1+ α 4 α we can ensure that 1) θ 2 α − 12 > 0 ; 2) α (1 − θ ) 1 − α ≥ α 2 .
Finally we have:
[Equation]
and
[Equation]
Given hypothesis class H , one can bound the growth function G H ( n ) through its VC-dimension.
Next we present the well known Sauer–Shelah Lemma below.
Lemma 5 (Sauer–Shelah Lemma(See
(Blum et al., 2016; Mohri et al., 2018; Sauer, 1972))) .
Let d vc ( H ) be the VC-dimension of hypothesis class H , for all n ∈ N ,
[Equation]
Now we are ready to prove the main result.
Theorem 4.
Under Assumption 1, given a set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process, hypothesis class H with VC-dimension d vc ( H ) , F with
VC-dimension d vc ( F ) and
[Equation]
If α ∈ (0 , 0 . 9) and:
[Equation]
We have with probability at least 1 − δ :
[Equation]
Proof.
I) above can be proved by observing c
= 1+ 1+
[Equation] [Equation]
and applying Theorem 3 with Sauer’s Lemma to bound the growth function in terms of VC-dimension.
Also notice that c = 1+ 1+ α
[Equation]
given α < 0 .
9 . II) can be proved by observing that I) implies P [ g ∗ S n ( x ) (cid:54) =
g ∗ ( x )]
= O ( ε ) and by invoking Theorem 1.
Remark 3.
Let us point out that our proposed selective strategy is different from the consistent selective strategy in (El-Yaniv et al., 2010).
Instead of rejecting by looking for consistent output from all hypothesis in the version space, our approach deals with one single reasonably accurate hypothesis (the empirical minimizer).
We leverage empirical mistakes made by the predictor in order to learn a selector, aiming to reject (only) the mistakes in a data driven manner.
This avoids dealing with the issues found in Theorem 14 in (El-Yaniv et al., 2010), where the selector fails to select any data points.
Remark 4.
In (Cortes et al., 2016), a second hypothesis for the selector is introduced and analyzed, and at the same time, multiple commonly used loss functions are scrutinized and generalization results are provided.
The major difference between this work and (Cortes et al., 2016; Geifman & El-Yaniv, 2019) is the motivation pertaining to selective learning.
While in (Cortes et al., 2016;
Geifman & El-Yaniv, 2019)
the selective loss is designed from a coverage ratio perspective, i.e. one wants to trade coverage ratio for a higher precision (selective loss), our approach is designed to distinguish data that is naturally unlearnable and unpredictable.
This difference leads to an alternative theoretical result.
While the analysis in (Cortes et al., 2016) focuses on selective risk, our theoretical analysis focuses on the quality of the selector in terms of both the false positive and false negative errors with the ﬁnal goal of rejecting uninformative data with high probability.
C T IGHTNESS ANALYSIS
In this section, we investigate whether it is necessary to have ˜ Ω (cid:16) d vc ( H )+ d vc ( F
[Equation]
2 (cid:15)(cid:17) samples to achieve
[Equation]
and
[Equation]
We will show that there exists a distribution D α among the Noisy Generative Process family and hypothesis class H , F such that achieving guarantees in Theorem 4 requires (cid:101) Ω( d vc ( H )+ d vc ( F ) α 2 (cid:15)) samples, thus making our analysis tight within logarithmic factor.
Note that our lower bound is for empirical minimizers of R ( f ) and W ( g, f, θ ) ; it only illustrates the tightness of our analysis, and thus is not an information theoretic bound for this family of problem - it remains unknown whether strategies other than the empirical minimization of R ( f ) and W ( g, f, θ ) can improve sample complexity to a rate of 1 αε , observing that the informative data is realizable, which usually only requires O ( 1 ε ) samples for an (cid:15) statistical error.
Our construction leverages the idea from (Ehrenfeucht et al., 1989) about the support of the informative data.
Over the support of uninformative data, noise buries the signal of the informative data, thus misleading the empirical minimizer.
In our analysis, we will leverage the anti-concentration results of the Chernoff bound, with the proof idea that comes from (Klein & Young, 2015):
Lemma 6 (tightness of the Chernoff bound) .
Let X be the average of k independent, 0/1 random variables (r.v.).
For any (cid:15) ∈ (0 , 1 / 2] and p ∈ (0 , 1 / 2] , assuming (cid:15)pk ≥ 6 , pk ≥ 6 , ε ≤ 13 , we have:
• If each r.v. is 1 with probability p , then
[Equation]
• If each r.v. is 1 with probability p , then
[Equation]
Proof.
The proof is similar to Lemma 5.2 in (Klein & Young, 2015) with a different choice of parameters.
With Stirling’s approximation, i ! =
√ 2 πi ( i/e ) i e λ with λ ∈ [1 / (12 i + 1) , 1 / 12 i ] one can show:
[Equation]
Also note: P [ X ≤ (1 − (cid:15) ) p ] = (cid:80) (1 −
[Equation]
) p
[Equation]
=
P [ X = ik ] .
Let (cid:96) =
(cid:98) (1 − 2 (cid:15) ) pk (cid:99) .
Given the fact that εpk ≥ 6 , we have (1 − 2 (cid:15) ) pk − 1 ≤ (cid:96) ≤ (1 − 2 (cid:15) ) pk.
It sufﬁces to lower bound (cid:80) (1 − ε ) p i =(1 − 2 ε )
p P
[ X = ik ] which is at least εpk P
[ X = (cid:96)k ] .
From Equationwe know that we need to bound A = 2 e (cid:15)pk/ √ 2 π(cid:96) and B = (cid:0) k(cid:96) (cid:1) (cid:96) (cid:0) kk − (cid:96) (cid:1) k − (cid:96) p (cid:96) (1 − p ) k − (cid:96) .
For term
A , since εpk ≥ 6 , l ≤ (1 − 2 ε ) pk thus we need pk ≥ 9 e − 2 ε ε to get 2 ε √ pk e √ 2 π (1 − 2 ε ) ≥
e − ε 2 pk .
Since ε ≤ 13 , it sufﬁces to have pk ≥ 16 .
To bound B we need to show:
[Equation]
Since (cid:0) k(cid:96) (cid:1) (cid:96) p (cid:96) ≥ (cid:16) 1 (1 − 2 ε ) (cid:17) l and (1 − p ) k − l (cid:16) kk − l (cid:17) k
[Equation]
l = (cid:16) (
[Equation]
(cid:17) k − l we have:
[Equation]
We consider the following distribution.
Let d vc ( F ) =
d and e j be the standard basis with jth coordinate equal to 1 and 0 otherwise.
We set Ω j U
=
+2 , − 2 } · e j , Ω U = ∪ dj
=1 Ω j U and Ω jI = { +1 , − 1 } · e j , Ω I = ∪ dj
=1 Ω jI .
Let F = sign ( w (cid:62) x ) and H = 2 (cid:81) ni
=1 g j ( x ) − 1 , where g j ( x ) ∈ { 1 {| e (cid:62) j x | ≥ c } ∪ { 0 } , 1 {| e (cid:62) j x | ≤ c }} .
By Lemma 3.2.3 in (Blumer et al., 1989)
we know d vc ( H ) ≤ 6 d log( d ) .
We construct D α as follows: in Ω U , each standard basis has 1 − αd mass with sign generated via coin ﬂipping (fair).
Having generated x , y ∈ { +1 , − 1 } is generated via a coin ﬂipping.
In Ω I , we put α (1 − ε ) mass on e 1 and αεd − 1 mass on other standard basis e j , with sign generated via a (fair) coin ﬂipping.
Having generated x , y ∈ { +1 , − 1 } is generated via sign( w ∗(cid:62) x ) where w ∗ denotes a vector of ones.
The above construction allows one to calculate the expected error by counting the number of basis that are not learned correctly, i.e., each j ∈ [ d ] s.t. w j < 0 implies an error αε 2( d − 1) in expectation.
Since:
[Equation]
by Markov inequality we have :
[Equation]
Also note that E x ∼D α [ 1 { x ∈ Ω j 1 } 1 { x = − 2 e j } ] = 1 − α 2 d .
In order to derive the probability that w j < 0 via empirical minimization, we need to bound the following probability:
[Equation]
To do so, we consider the following n j random variables that are conditioned on x ∈ Ω j 1 : z i =if e (cid:62)
j x i < 0 ,
y i = 1 or e (cid:62) j x i ≥ 0 ,
y i = − 1 and z i = 0 otherwise.
One can see 2 (cid:80)
i
[Equation]
n j z i − n j = (cid:80) x i ∈
Ω j ,e (cid:62) j x i <
y i
[Equation]
(cid:80) x i ∈
Ω j ,
[Equation]
j x i >
y i .
Thus :
[Equation]
If n ≤ d 432 ε 2 α 2 , we have for each j , with probability at least 0 .
40 , w j <
due to the empirical minimization strategy.
Each w j < 0 contributes αε 2( d − 1) risk in expectation.
With probability 0 .
2 , 20% of coordinates can not be learned correctly, thus resulting in a risk of at least 0 .
1 εα in expectation.
Next we show that the selector is going to suffer from a poorly learned classiﬁer in terms of its false negative rate.
Now lets focus on the case when E ( x,y )
∼D α [ f S n ( x ) (cid:54) =
y ] ≥ 12 (1 − α ) + 0 .
1 αε (which happens with probability at least 0 . 2 ), we will show that E x
∼D α [ 1 { g S n ( x ) (cid:54) =
g ∗ ( x ) }| g ∗ ( x ) > 0] ≥ 0 .
1 ε .
Let J 1 be the set of j ’s s.t.
w j < 0 and J 2 for w j > 0 .
Suppose θ > α , and consider empirical minimization:
[Equation]
For j (cid:48) ∈ J
[Equation]
, g ∈ arg
[Equation]
S
[Equation]
( f
[Equation]
S n , g, θ ) if g j
[Equation]
( x ) = 1 {| e j
[Equation]
(cid:62) x | ≥ c } , c > 2 (and for other basis j , g j can be arbitrary).
For j (cid:48) ∈ J 2 , g ∈ arg min g ∈H W S
n ∩ Ω j (cid:48)
( f ∗ S n , g, θ ) if g j (cid:48)
( x ) = 1 {| e j (cid:48) (cid:62) x | ≤ c } , c ∈ [1 , 2) .
Now consider g ∗ S n = 2 (cid:81) dj
=1 g j ( x ) − 1 , g j ( x ) = 1 {| e (cid:62) j x | ≥ c } with c > 2 if j ∈ J 1 and g j ( x ) = 1 {| e (cid:62) j x | ≤ c } , c ∈ [1 , 2) if j ∈ J 2 .
One can see that such g ∗ S n is in the intersection of set of minimizers of W S n ∩ Ω j ( f ∗ S n , g, θ ) thus attains the minimum.
The total mass of x s.t.
g ∗ S n ( x ) (cid:54) =
g ∗ ( x ) is at least |J 1 | αε d − 1 since for all x ∈ Ω j with j ∈ J 1 , g ∗ S n ( x ) (cid:54) =
g ∗ ( x ) .
The event E : E ( x,y )
∼D α [ f S n ( x ) (cid:54) =
f ∗ ( x )] ≥ 12 (1 − α )
+ 0 .
1 αε happens with probability at least 0 .which implies that |J 1 | ≥ 0 .
2 d.
Thus we have following lower bound on the False Negative:
[Equation]
D S UPPLEMENTAL M ATERIAL FOR E MPIRICAL S TUDY
Extension to Multi-class.
Our method extends to multi-class setting naturally.
In the case of K -class classiﬁcation, we predictors f ( x ) =
f
( x ) 1: K : X → ∆ K where ∆ K is the K -simplex.
In multi-class scenario, our selector loss remains the same.
Meanwhile, we use multi-class cross entropy loss to train classiﬁer.
The pseudo-informative label becomes z i = 1 { arg max k ∈
[ K ] f ( x i ) k
= y i } .
Experiment setting with Unknown α (More Details).
For MNIST/Fashion-MNIST experiment, the backbone model we use for each candidate is TinyCNN, which is a CNN with two consecutive convolution layers and 3 consecutive fully connected layers.
This architecture provides enough model capacity for an MNIST-type dataset.
We use 196 as the batch size.
We use Adam as the optimizer with learning rate 1e-3 and weight decay 1e-4.
We train with each algorithm for 162 epochs.
We use the default hyper-parameters for every method (i.e., internal selective learning-speciﬁc defaults, as reported in (Geifman & El-Yaniv, 2019;
Liu et al., 2019), and β = θ (1 − α ) α (1 − θ ) for our algorithm).
It simulates a practical scenario where hyper-parameter optimization is impossible, since the ground truth regarding which datapoints are actually learnable is never revealed.
For the SVHN experiment, the backbone model we use is ResNet18 (He et al., 2016b) for every candidate.
We use 128 as the batch size.
We use Adam as the optimizer with learning rate 1e-3 and weight decay 1e-4.
We train each algorithm for 162 epochs.
Weight of Informative data.
From Figure 3, we can see that our algorithm can pick out almost all informative data by the end of training for all informative/uninformative ratios even without knowing the ground-truth α . (a) MNIST. Weight on Informative Data (b) SVHN.
[Figure]
Experiments with Known α .
We also test the performance of each method in the idealized case.
In this experiment, we give the groud-truth α to each baseline and perform an exhaustive HPO (Hyper Parameter Optimization).
We use the same backbone, optimizer and the number of training epochs as before.
We run a random hyper-parameter search 500 times (selective loss weight and parameter λ for SelectiveNet and parameter o in DeepGambler) and record the best performance of each baseline.
In this setting, our method still outperform baselines for most levels of noise ratios.
In this experiment, we use two stage training for our algorithm.
We ﬁrst run 120 epochs to ﬁnish 1st round data selection and then run 42 epochs to train a classiﬁer only with these selected data.
The performance is measured with this reﬁned the classiﬁer.
Experiment with Low Ratio of Uninformative Data.
We also conduct experiments with datasets that contain minority stochastic data.
This experiment is conducted under both practical (unknown α ) and idealized scenario (known α ).
In practical setting (unknown α ), our proposed algorithm still wins by a large margin.
In the case when α is known, DeepGambler outperforms other methods in most cases.
We hypothesize that when noisy data minority, our method doesn’t get enough samples to fully learn the pattern of uninformative data [Table]
D. E XPERIMENT WITH K NOWN α
[Table]
[Table]
[Table]
[Table]
[Table]
Results on low noise level for both MNIST and SVHN - Known α .
We evaluate the methods over different fractions of informative data
[Table]
D. E XPERIMENT WITH L OW R ATIO OF U NINFORMATIVE D ATA D. I MPLEMENTATION D ETAIL
For all our experiments, we are using NVIDIA Tesla V100 GPUs, and we use AWS Sagemaker service for our HPO processes.
We repeat each experiment 5 times with random seed 77, 78, 79, 80, 81.
For MNIST/Fashion-MNIST experiment where α is unknown, the training and testing data set split is listed in Table 11.
The backbone we use is TinyCNN (in section 4).
We use 196 as our batch size and train for 162 epochs for all listed methods.
We use Adam as the optimizer with learning rate 1eand weight decay 1e-4.
We shrink our learning rate at epoch 45 and 90 by half each time.
For SVHN experiment where α is unknown, the training and testing data set split is listed in Table 11.
The backbone we use is ResNet18.
We use 128 as our batch size and train for 162 epochs for all listed methods.
We use Adam as the optimizer with learning rate 1e-3 and weight decay 1e-4.
We shrink our learning rate at epoch 45 and 90 by half each time.
For both experiments where α is known
[Figure]
and HPO is enabled, we maintain the same backbone and optimizer.
The hyper-parameter setting is documented in the code.
The code will be open sourced together with the publication of the paper.
E E XTENTION OF T HEOREM 4 IN THE CASE WHERE ONLY AN UPPER BOUND ON α IS GIVEN
Theorem 5.
Under Assumption 1, given set of samples S n = { ( x 1 , y 1 ) , ..., ( x n , y n ) } drawn i.i.d.
from the Noisy Generative Process with α ∈ (0 , 1 − ξ ] , ξ ≥ 0 . 2 , hypothesis class H with VC-dimension d vc ( H ) , F with VC-dimension d vc ( F ) and f ∗ S n = arg min f ∈F R S
( f ) , g S n
[Equation]
= arg min g ∈H W S n (cid:0) g, f ∗ S n , θ (cid:1) .
{+If θ ∈ (cid:16) 
+}
α 72 , min { 1 + ξ 2(1 − ξ ) , 10 } α (cid:17) and n ≥ 3200 [ ( d vc ( H )+ d vc ( F ))log( 1 ε )+log( 16 δ ) ] (cid:15) 2 α.
We have with probability at least 1 − δ :
I)
P [ g ∗ S n ( x ) ≥ 0 | g ∗ ( x ) ≤ 0] =
O ( (cid:15) )
[ False Positive ]
P [ g
[Equation]
S n ( x ) < 0 | g
[Equation]
( x ) ≥ 0] ≤ O ( (cid:15) )
[ False Negative ] (39)
II)
P
[Equation]
( (cid:15) )
III) Further more, to obtain g S n ∗ =
arg min g ∈H W S n (cid:0) g, f ∗ S n , θ (cid:1) it sufﬁces to minimize following loss with β ∈ (1 + 1 73 ξ − 1 , min { 1 + ξ 2(1 − ξ ) , 10 } ) :
Loss ( g ; f ∗ S n , S n , β ) = n (cid:88) i = (cid:8) β
[Equation]
Proof.
I) above can be proved by picking c ∈ ( 12 + 1144 , 12 + min { 92 , ξ 4(1 − ξ ) } ) in Theorem 3.
Note this range is valid since α ≤ 1 − ξ , we have
[Equation]
By a procedure similar to Theorem 4 (using the Sauer’s Lemma) we have I ) .
Note c ∈ ( 12 + 1144 , 12 + min { 1 , ξ 4(1 − ξ ) } )
⇐⇒ θ ∈ (cid:16) 73 α 72 , min { 1 + ξ 2(1 − ξ ) , 2 } α (cid:17) .
II) can be proved by observing that I) implies P [ g ∗ S n ( x ) (cid:54) =
g ∗ ( x )]
= O ( ε ) and by invoking Theorem 1.
Next we prove III).
It is easy to verify that the minimizer of W S n ( g, f ∗ S n ) is equivalent to minimizer of Loss ( g ; f ∗ S n , S n , β ) (by replacing β = θ (1 − α ) α (1 − θ ) ).
{+It sufﬁces to analyze that β ∈ (1+ 1 73 ξ − 1 , min { 1+ ξ 2(1 − ξ ) , 10 } ) implies that θ ∈ (cid:16) 
+}
α 72 , min { 1 + ξ 2(1 − ξ ) , 10 } α (cid:17) .
First note that x 1 − x is monotone increasing in (0 , 1) , it sufﬁces to compute image of θ (1 − α )
α (1 − θ ) and check if β ∈ (1 + 1 73 ξ − 1 , min { 1 + ξ 2(1 − ξ ) , 10 } ) stays in such range.
It can be easily veriﬁed that α ≤ 1 − ξ implies the lower bound 1 + 1 73 ξ − 1 and α > 0 implies the upper bound 1 + ξ 2(1 − ξ ) .
F A BLATION S TUDY ON H YPERPARAMETERS
We provide an ablation study where we vary the hyper-parameter in each baseline.
For SelectiveNet, we ﬁrst ﬁx λ to be 32 (default setting that is recommended by the author in the original paper)
and then we vary a from 0.1 to 0.7.
Then we ﬁx a to be 0.5 (default setting) and then we vary λ from 1 to 66.
For DeepGambler, we vary o from 1 to 7.
Finally, for our algorithm, we increase the hyper-parameter β 4 to 10.
We can see that the performance of all baselines are quite sensitive to the choice of hyper-parameter and will experience some large ﬂuctuation.
On the contrast, our algorithm is stable against the choice of hyper-parameter.
This empirical observation supports that there exists a wide range of β so that one can control the FNR and FPR if α is bounded away from 1 as it is implied in Theorem 5.
Furthermore, in all scenario, our algorithm’s performance is better than these two baselines in two sense.
On one hand, our selector has better precision such that we can recover almost all informative data while the other two cannot.
These two baseline tend to to select the whole data set indistinguishably (low precision and high recall).
On the other hand, these baselines show consistently deteriorated risk performance than ours because of their selector fails to pick informative data.
[Figure]
[Figure]
[Figure]