O N THE M INIMAL S UPERVISION FOR T RAINING A NY B INARY C LASSIFIER FROM O NLY U NLABELED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classiﬁcation.
In this paper, we study training arbitrary (from linear to deep) binary classiﬁer from only unlabeled (U) data by ERM.
We prove that it is impossible to estimate the risk of an arbitrary binary classiﬁer in an unbiased manner given a single set of U data but it becomes possible given two sets of U data with different class priors .
These two facts answer a fundamental question—what the minimal supervision is for training any binary classiﬁer from only U data.
Following these ﬁndings, we propose an ERM-based learning method from two sets of U data and then prove it is consistent .
Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data
I NTRODUCTION
With some properly chosen loss function (e.g., Bartlett et al., 2006; Tewari & Bartlett, 2007; Reid &
Williamson, 2010) and regularization (e.g., Tikhonov, 1943; Srivastava et al., 2014), empirical risk minimization (ERM) is the common practice of supervised classiﬁcation (Vapnik, 1998).
Actually,
ERM is used in not only supervised learning but also in weakly-supervised learning .
For example, in semi-supervised learning (Chapelle et al., 2006), we have very limited labeled (L) data and a lot of unlabeled (U) data, where L data share the same form with supervised learning.
Thus, it is easy to estimate the risk from L data, and ERM can be done with regularization based on U data (including but not limited to Grandvalet & Bengio, 2004; Belkin et al., 2006; Mann & McCallum, 2007;
Niu et al., 2013; Miyato et al., 2016; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Luo et al., 2018).
Nevertheless, L data may differ from supervised learning in not only the amount but also the form.
For instance, in positive-unlabeled learning (Elkan & Noto, 2008; Ward et al., 2009), all L data are from the positive class, and due to the lack of L data from the negative class it becomes impossible to estimate the risk from only L data.
To this end, a two-step approach to ERM has been considered (du Plessis et al., 2014; 2015; Niu et al., 2016; Kiryo et al., 2017).
Firstly, the risk is rewritten into an equivalent expression, such that it just involves the same distributions from which L and U data are sampled—this step leads to certain risk estimators.
Secondly, the risk is estimated from both L and U data, and the resulted empirical training risk is minimized (e.g. by Robbins & Monro, 1951;
Kingma & Ba, 2015).
In this two-step approach, U data are used for risk evaluation that is the core of ERM and is mandatory in ERM, and hence risk rewrite enables ERM and is the key of success.
One step further from positive-unlabeled learning is learning from only U data without any L data.
This is signiﬁcantly harder than previous learning problems (cf.
Figure 1).
However, we would still like to train arbitrary binary classiﬁer , in particular, deep networks (Goodfellow et al., 2016).
As a consequence, we prefer ERM to clustering methods (e.g., Xu et al., 2004; Gomes et al., 2010).
The critical point is how to estimate the risk from only U data, and our solution is again ERM-enabling risk rewrite in the aforementioned two-step approach.
The ﬁrst step should lead to an unbiased risk estimator that will be used in the second step.
Subsequently, we can evaluate the empirical training and/or validation risk by plugging only U training/validation data into the risk estimator.
Thus, this two-step ERM needs no L validation data for hyperparameter tuning , which is a huge advantage in training deep models nowadays.
Note that given only U data, by no means could we learn the class priors (Menon et al., 2015), so that we assume all necessary class priors are also given .
This is the
−−  4 (a) P component −−   −−  4 (b) N component −−   −−  4 (c) U set1 −−
−−  4 (d) U set2 −−   −−−−    4 −−−−
CCN
UU-biased
UU
Oracle unique type of supervision we will leverage throughout this paper, which implies that the problem of interest still belongs to weakly-supervised learning rather than unsupervised learning.
In this paper we raise a fundamental question in weakly-supervised learning—how many sets of U data with different class priors are necessary for rewriting the risk?
Our answer has two aspects:
• Risk rewrite is impossible given a single set of U data (see Theorem 2 in Sec. 3); • Risk rewrite becomes possible given two sets of U data (see Theorem 4 in Sec. 4).
This suggests that three class priors 1 are all you need to train deep models from only U data, while any two 2 should not be enough.
The impossibility is a proof by contradiction, and the possibility is a proof by construction, following which we explicitly design an unbiased risk estimator.
Therefore, with the help of this risk estimator, we propose an ERM-based learning method from two sets of U data.
Thanks to the unbiasedness of our risk estimator, we derive an estimation error bound which certainly guarantees the consistency of learning (Mohri et al., 2012; Shalev-Shwartz & Ben-David, 2014).
3 Experiments demonstrate that the proposed method could train multilayer perceptron, AllConvNet (Springenberg et al., 2015) and ResNet (He et al., 2016) from two sets of U data; it could outperform state-of-the-art methods for learning from two sets of U data.
See Figure 1 for how the proposed method works on a Gaussian mixture of two components.
P ROBLEM SETTING AND RELATED WORK
Consider the binary classiﬁcation problem.
Let X
and Y be the input and output random variables such that p ( x, y ) is the underlying joint density , p p ( x )
=
p ( x | Y = +1) and p n ( x ) = p ( x | Y = − 1) are the P and N class-conditional densities , p ( x ) is the marginal density , and
π p = p ( Y = +1) is the class-prior probability .
Data generation process Let θ and θ (cid:48) be two class priors such that θ (cid:54) =
θ (cid:48) , and let
[Equation]
1) be the marginal densities from which U training data are drawn.
Eq.
(1) implies there are p tr ( x, y ) and p (cid:48)
tr ( x, y ) , whose class-conditional densities are same and equal to those of p ( x, y ) , and whose class priors are different, i.e.,
[Equation]
If we could sample L data from p tr ( x, y ) or p (cid:48) tr ( x, y ) , it would reduce to supervised learning under class-prior change (Quiñonero-Candela et al., 2009).
Nonetheless, the problem of interest belongs to weakly-supervised learning—U training (and validation) data are supposed to be drawn according to (1).
More speciﬁcally, we have
[Equation]
where n and n (cid:48) are two natural numbers as the sample sizes of X tr and X (cid:48) tr .
This is exactly the same as du Plessis et al.
and
Menon et al.
(2015) with some different names.
In Menon et al.
(2015), θ and θ (cid:48) are called corruption parameters , and if we assume θ > θ (cid:48) , p tr ( x ) is called the corrupted
P density and p (cid:48)
tr ( x ) is called the corrupted N density .
Despite the same data generation process in (2), a vital difference between the problem settings is performance measures to be optimized.
Performance measures Let g : R d → R be an arbitrary decision function , i.e., g may literally be any binary classiﬁer.
Let (cid:96) :
R → R be the loss function , such that the value (cid:96) ( z ) means the loss by predicting g ( x ) when the ground truth is y where z = yg ( x ) is the margin .
The risk of g is
[Equation]
3) where E p [ · ] means E
X ∼ p p [ · ] and E n [ · ] means E
X ∼ p n [ · ] respectively.
If (cid:96) is the zero-one loss that is deﬁned by (cid:96) 01 ( z ) =
(1 − sign( z )) / 2 , the risk is named the classiﬁcation error that is the standard performance measure in classiﬁcation problems.
A balanced version of (3) is
[Equation]
and if (cid:96) is (cid:96) 01 , (4) is named the balanced error (Brodersen et al., 2010).
The vital difference is that (3) is chosen in the current paper whereas (4) is chosen in du Plessis et al.
and
Menon et al.
(2015) as the performance measure to be optimized.
We argue that (3) is more natural than (4) as the performance measure for binary classiﬁcation.
By the phrase “binary classiﬁcation”, we mean π p is neither very large nor very small.
Otherwise, due to extreme values of π p (i.e., either π p ≈ 0 or π p ≈ 1 ), the problem under consideration should be retrieval or detection rather than binary classiﬁcation.
Note that B ( g ) (cid:54) =
R ( g ) unless π p = 12 since g is arbitrary, which implies that (4) is misleading so long as (3) is the performance measure.
Related work Learning from only U data is previously regarded as discriminative clustering (Xu et al., 2004; Valizadegan & Jin, 2006; Li et al., 2009; Gomes et al., 2010; Sugiyama et al., 2014;
Hu et al., 2017).
Their goals are to maximize the margin or the mutual information between X and Y .
As a result, they rely on the cluster assumption (Chapelle et al., 2002) and the assumption that one class corresponds to exactly one cluster .
The second assumption is rarely satisﬁed in practice.
As mentioned earlier, learning from two sets of U data is already studied in du Plessis et al.
and Menon et al. (2015).
Both of them adopt (4) as the performance measure.
In the former paper, g is learned by estimating sign( p tr ( x ) − p (cid:48) tr
( x )) .
In the latter paper, g is learned by taking noisy L data from p tr ( x ) and p (cid:48) tr
( x ) as clean L data from p p ( x ) and p n ( x ) , and then its threshold is moved to the correct value by post-processing.
In summary, instead of ERM, they evidence the possibility of empirical balanced risk minimization , and no impossibility is proven.
Our ﬁndings are compatible with learning from label proportions (Quadrianto et al., 2009; Yu et al., 2013).
In Quadrianto et al.
(2009), it is proven that the minimal number of U sets equals the number of classes.
However, their ﬁnding only holds for the linear model, the logistic loss, and their proposed method based on mean operators.
On the other hand, Yu et al.
(2013) is not ERM-based; it is based on discriminative clustering together with expectation regularization (Mann & McCallum, 2007).
At ﬁrst glance, our data generation process, using the names from Menon et al.
(2015), looks quite similar to that of learning with noisy labels (cf.
Natarajan et al., 2013).
Actually, these two are fairly different, and the differences are reviewed and discussed in Menon et al.
and van Rooyen &
Williamson (2018).
Along this line of research, just a few papers explore instance-dependent noise models (Menon et al., 2016; Cheng et al., 2017), and the vast majority of papers employ instanceindependent noise models (e.g., Natarajan et al., 2013; Sukhbaatar et al., 2015; Menon et al., 2015;
Liu & Tao, 2016; Goldberger & Ben-Reuven, 2017; Patrini et al., 2017; Han et al., 2018a) or have no noise model but many heuristics (e.g., Reed et al., 2015; Jiang et al., 2018; Ren et al., 2018; Han et al., 2018b).
There exist two instance-independent noise models: class-conditional noise (CCN) in Angluin & Laird (1988) and mutually contaminated distributions (MCD) in Scott et al.
Denote by ˜ y and ˜ p ( · ) the corrupted label and distributions.
Then, CCN and MCD are deﬁned by
[Equation]
and
[Equation]
where both of T CCN and T MCD are 2-by-2 matrices but T CCN is column normalized and T MCD is row normalized.
It has been proven in Menon et al.
(2015) that CCN is a strict special case of MCD.
To be clear, ˜ p (˜ y ) is ﬁxed in CCN once ˜ p
(˜ y
| x ) is speciﬁed while ˜ p (˜ y ) is free in MCD after ˜ p ( x | ˜ y ) is speciﬁed.
Furthermore, ˜ p ( x ) = p ( x ) in CCN but ˜ p ( x ) (cid:54) = p ( x ) in MCD.
Due to this covariate shift ,
CCN methods do not ﬁt MCD problem setting, though MCD methods ﬁt CCN problem setting.
To the best of our knowledge, the proposed method is the ﬁrst MCD method based on ERM.
L EARNING FROM ONE SET OF U DATA
From now on, we prove that knowing π p and θ is insufﬁcient for rewriting R ( g ) .
3. A BRIEF REVIEW OF ERM
To begin with, we review ERM (Vapnik, 1998) by imaging that we are given X p = { x 1 , . . .
, x n } ∼ p p ( x ) and X n = { x (cid:48) 1 , . . .
, x (cid:48) n (cid:48) } ∼ p n ( x ) .
Then, we would go through the following procedure:
Choose a surrogate loss (cid:96) ( z ) , so that R ( g ) in Eq.
(3) is deﬁned.
Choose a model G , so that min g ∈G R ( g ) is achievable by ERM. 3.
Approximate R ( g ) by
[Equation]
Minimize (cid:98) R pn ( g ) , with appropriate regularization, by favorite optimization algorithm.
Here, (cid:96) should be classiﬁcation-calibrated (Bartlett et al., 2006), in order to guarantee that R ( g ; (cid:96) ) and R ( g ; (cid:96) 01 ) have the same minimizer over all measurable functions.
This minimizer is the Bayes optimal classiﬁer and denoted by g ∗∗ = arg min g R ( g ) .
The Bayes optimal risk R ( g ∗∗ ) is usually unachievable by ERM as n, n (cid:48)
→ ∞ .
That is why by choosing a model G , g ∗ =
arg min g ∈G R ( g ) is changed as the target to which (cid:98) g pn = arg min g ∈G (cid:98) R pn ( g ) converges as n, n (cid:48) → ∞ .
In statistical learning, the approximation error is R ( g ∗ ) − R ( g ∗∗ ) , and the estimation error is R ( (cid:98) g pn ) − R
( g ∗ ) .
Learning is consistent if and only if the estimation error converges to zero as n, n (cid:48)
→ ∞ .
3. I MPOSSIBILITY OF RISK REWRITE
Recall that R ( g ) is approximated by (5) given X p and X n , which does not work given X tr and X (cid:48)
tr .
We might rewrite R ( g ) such that it can be approximated given
X tr and/or X (cid:48) tr .
This is known as the backward correction (Natarajan et al., 2013; Patrini et al., 2017) in learning with noisy labels.
Deﬁnition 1.
We say that R ( g ) in (3) is rewritable given p tr , if and only if 5 there exist constants a and b , such that for any g it holds that
[Equation]
where E p tr [ · ] means E X ∼ p tr [ · ] and ¯ (cid:96) ( z )
=
a(cid:96) ( z ) + b(cid:96) ( − z ) is the corrected loss function.
Theorem 2.
Let (cid:96) be (cid:96) 01 , or any bounded surrogate loss satisfying that
[Equation]
[Equation]
Assume p p and p n are almost surely separable and θ is arbitrary.
Then, R ( g ) is not rewritable.
This theorem shows that under the separability assumption of p p and p n , R ( g ) is not rewritable.
As a consequence, we lack a learning objective, that is, the empirical training risk.
It is even worse—we cannot access the empirical validation risk of g after it is trained by other learning methods such as discriminative clustering.
In particular, (cid:96) 01 satisﬁes (7), which implies that the common practice of hyperparameter tuning is disabled by Theorem 2, since U validation data also follow p tr .
L EARNING FROM TWO SETS OF U DATA
From now on, we prove that knowing π p , θ and θ (cid:48) is sufﬁcient for rewriting R ( g ) .
4. P OSSIBILITY OF RISK REWRITE , AND UNBIASED RISK ESTIMATORS
We have proven that R ( g ) is not rewritable given p tr , and Quadrianto et al.
(2009) has proven that
R ( g ) can be estimated from X tr and X (cid:48) tr , where g is a linear model and (cid:96) is the logistic loss.
These facts motivate us to investigate the possibility of rewriting R ( g ) , where g and (cid:96) are both arbitrary.
Deﬁnition 3.
We say that R ( g ) is rewritable given p tr and p (cid:48) tr , if and only if 8 there exist constants a , b , c and d , such that for any g it holds that
[Equation]
where ¯ (cid:96) + ( z ) =
a(cid:96) ( z ) + b(cid:96) ( − z ) and ¯ (cid:96) − ( z ) =
c(cid:96) ( z ) + d(cid:96) ( − z ) are the corrected loss functions.
Theorem 4.
Assume θ and θ (cid:48) are arbitrary but satisfy θ > θ (cid:48) ; otherwise, swap p tr and p (cid:48) tr to make sure θ > θ (cid:48) .
Then, R ( g ) is rewritable, by letting
[Equation]
Theorem (4) immediately leads to an unbiased risk estimator, namely
[Equation]
Eq.
(10) is useful for both training (by plugging U training data into it) and hyperparameter tuning (by plugging U validation data into it).
We hereafter refer to the process of obtaining the empirical risk minimizer of (10), i.e., (cid:98) g uu = arg min g ∈G (cid:98) R uu ( g ) , as unlabeled-unlabeled (UU) learning .
The proposed UU learning is by nature ERM-based, and consequently (cid:98) g uu can be obtained by powerful stochastic optimization algorithms (e.g., Duchi et al., 2011; Kingma & Ba, 2015).
Simpliﬁcation Note that (10) may require some efforts to implement.
Fortunately, it can be simpliﬁed by employing (cid:96) that satisﬁes a symmetric condition :
[Equation] [Equation]
Eq.
(11) covers (cid:96) 01 , the ramp loss (cid:96)
[Equation]
z [Equation] 1 − z ) / 2 }} (du Plessis et al., 2014;
Niu et al., 2016) and the sigmoid loss (cid:96) sig ( z )
= 1 / (1 + exp( z ))
(Kiryo et al., 2017).
With the help of (11), we can simplify (10) as
[Equation]
(12) where α = ( θ (cid:48)
+ π p − 2 θ (cid:48)
π p ) / ( θ − θ (cid:48) ) and α (cid:48) =
( θ + π p − 2 θπ p ) / ( θ − θ (cid:48) ) .
Just like (10), (12) is an unbiased risk estimator, and it is easy to implement in many deep learning frameworks by ordinary cost-sensitive learning.
Special cases Consider some special cases of (10) by specifying θ and θ (cid:48) .
It is obvious that (10) reduces to (5) for supervised learning, if θ = 1 and θ (cid:48)
= 0 .
Next, (10) reduces to
[Equation]
if θ = 1 and θ (cid:48)
=
π p , and we recover the unbiased risk estimator in positive-unlabeled learning (du
Plessis et al., 2015; Kiryo et al., 2017).
Additionally, (10) reduces to a fairly complicated unbiased risk estimator in similar-unlabeled learning (Bao et al., 2018), if
[Equation] [Equation] or vice versa.
Therefore, UU learning is a very general framework in weakly-supervised learning.
[Table]
[Figure]
4. C ONSISTENCY AND CONVERGENCE RATE
The consistency of UU learning is guaranteed due to the unbiasedness of (10).
In what follows, we analyze the estimation error R ( (cid:98) g uu ) − R ( g ∗ ) (see Sec.
3.1 for the deﬁnition).
To this end, assume there are C g > 0 and C (cid:96) >
such that sup g ∈G (cid:107) g (cid:107) ∞
≤ C g and sup
| z |≤ C g (cid:96) ( z ) ≤ C (cid:96) , and assume (cid:96) ( z ) is Lipschitz continuous for all | z | ≤ C g with a Lipschitz constant L (cid:96) .
Let R n ( G ) and R (cid:48) n (cid:48) ( G ) be the Rademacher complexity of
G over p tr ( x ) and p (cid:48) tr ( x ) (Mohri et al., 2012; Shalev-Shwartz &
Ben-David, 2014).
For convenience, denote by χ n,n (cid:48) = α/ √ n + α (cid:48) / √ n (cid:48) .
Theorem 5.
For any δ > 0 , let C δ
=
(cid:112) (ln 2 /δ ) / 2 , then we have with probability at least 1 − δ ,
[Equation] [Equation]
where the probability is over repeated sampling of data for training (cid:98) g uu .
Theorem 5 ensures that UU learning is consistent (and so are all the special cases): as n, n (cid:48) → ∞ ,
R ( (cid:98) g uu ) → R ( g ∗ ) , since R n ( G ) , R (cid:48) n (cid:48) ( G ) → 0 for all parametric models with a bounded norm such as deep networks trained with weight decay.
Moreover, R ( (cid:98) g uu ) → R ( g ∗ ) in O p ( χ n,n (cid:48) ) , where O p denotes the order in probability, for linear-in-parameter models and non-parametric kernel models in reproducing kernel Hilbert spaces with a bounded norm (Schölkopf & Smola, 2001).
E XPERIMENTS
In this section, we experimentally analyze the proposed algorithm on deep neural network models trained on various benchmark datasets.
The implementation is based on Keras.
5. B ENCHMARK EXPERIMENTS WITH NEURAL NETWORK MODELS
We ﬁrst illustrate the operation of the proposed unbiased risk estimator and compare it with 3 supervised baselines: small PN, PN oracle and small PN prior-shift, where small PN/PN oracle denotes fully supervised classiﬁcation with 10 % /100 % training data, and small PN prior-shift means supervised classiﬁcation with 10% training data under class-prior change. 9 Note that small PN and PN 9 PN
[Table]
[Figure]
[Figure]
oracle use fully supervised training data generated from the same distribution as the test data, which are extremely advantageous.
We test on the widely adopted benchmarks, MNIST, Fashion-MNIST, SVHN and CIFAR-10.
Table 1 summarizes the speciﬁcation of the datasets.
For each dataset, we draw equal amount of unlabeled training data for X tr and X (cid:48) tr from Eq.
(1), where two different pairs of training class priors are considered: (0 . 9 , 0 . 1) , (0 . 8 , 0 . 2) .
The test data is directly drawn from the original joint distribution p ( x, y ) for evaluating the performance of the trained models.
The model and optimizer for MNIST and Fashion-MNIST were 5-layer fully connected neural network (FC) and SGD (Robbins & Monro, 1951).
For SVHN and CIFAR-10, we used 12layer all convolutional net (Springenberg et al., 2015) and 32-layer residual network (ResNet) (He et al., 2016) respectively, and the resulting objectives were minimized by Adam (Kingma & Ba, 2015).
We compared the performance of the proposed risk estimator Eq.
(10) with logistic loss (cid:96) log ( z ) =
log (1 + exp( − z ))
(Natarajan et al., 2013) and the simpliﬁed version Eq.
(12) with (cid:96) sig in Appendix C.2.
The results show that the risk estimators with (cid:96) sig and (cid:96) log perform similarly.
For simplicity, we select (cid:96) sig as the surrogate loss for training and 0-1 loss for testing in the following experiments.
More details on experimental setup can be found in Appendix C.1.
The experimental results are reported in Figure 2, where means and standard deviations of test risks based on the same 10 random samplings are shown, and a table of ﬁnal test risk is provided in Appendix C.2.
In the case of ( θ, θ (cid:48) )
= (0 . 9 , 0 . 1) (top row in Figure 2), we can see the proposed UU classiﬁers are comparable to PN oracle in most cases.
Note that for SVHN, the test class prior is only 0 . 27 , which is farther from the training class priors of UU.
Thus this setting is more advantageous for PN baselines without class prior shift but more challenging for UU and small PN prior-shift.
The results on SVHN show that UU still outperforms small PN while small PN prior-shift deteriorates severely.
For the harder case of ( θ, θ (cid:48) )
= (0 . 8 , 0 . 2) (bottom row in Figure 2), the performances of UU classiﬁers drop slightly, but are still better than small PN prior-shift.
The drop here can be explained by larger noise in the U sets when moving θ and θ (cid:48) closer, and we investigate this issue in Figure 3.
Analysis of moving θ and θ (cid:48) closer It is intuitive that if θ and θ (cid:48) are closer, the U sets will be less informative.
To investigate the inﬂuence of this, we conducted additional experiments on MNIST by moving θ and θ (cid:48) closer, where θ ∈ { 0 . 9 , 0 . 8 } and θ (cid:48) is gradually moved from 0 .
1 to 0 .
The experimental setup is exactly the same as before.
We reported the means and standard deviations of the test risks based on the same 10 random samplings in Figure 3.
The results show that the proposed unbiased UU method works reasonably well, while the performance of CCN drops severely.
It is because the marginal p ( x ) is shifted more distant from training to test stages as θ (cid:48) moves closer to θ , which will make the CCN-based risk estimator more biased.
Thanks to the unbiasedness of the proposed risk estimator, UU classiﬁers can handle the covariate shift reasonably well.
We further analyzed the inﬂuence of covariate shift by changing the sample sizes of the U sets in Appendix C.2, the experimental results are consistent.
Robustness of noisy training class priors In the above experiments, we assume the true training class priors θ and θ (cid:48) are exactly accessible, but in practice we may only be able to approximately specify them.
This motivates our study to try some cases when θ and θ (cid:48) are misspeciﬁed, in order to simulate UU learning in the wild.
We run more experiments by replacing the true training class priors ( θ, θ (cid:48) ) with ( ϑ, ϑ (cid:48) )
=
( (cid:15)θ, (cid:15) (cid:48) θ (cid:48) ) and give ( ϑ, ϑ (cid:48) ) to the learning method.
The experimental setup is exactly the same as before except that the training class priors are noisy.
We reported the means and standard deviations of the classiﬁcation error of the learned models based on the same 10 random samplings in Table 2.
The results show that the proposed method is fairly robust to the misspeciﬁcation of training class priors so long as | ϑ − θ | + | ϑ (cid:48) − θ (cid:48) | (cid:28) | θ − θ (cid:48) | .
5. C OMPARISON WITH STATE - OF - THE - ART METHODS
We ﬁnally compare our method with two state-of-the-art methods for dealing with two sets of U data: the proportion-SVM method (pSVM) (Yu et al., 2013) and the balanced error minimisation method (BER) (Menon et al., 2015).
We downloaded the codes from the webpage of authors.
11 Note that the pSVM method is based on maximum margin clustering (Xu et al., 2004; Valizadegan & Jin, 2006; Li et al., 2009) and the original codes of BER implement neural network by Matlab and use the second order optimization method.
Considering the time for 10 runs of each experiment, we use
UCI benchmarks and USPS datasets for the experiment following pSVM and BER.
By re-sampling the original datasets, we test several different settings of class prior π p .
The 5-layer FC and SGD were again used for training UU classiﬁer here.
For fairness, we also implemented BER method using the same network architecture and optimizer as UU (BER-FC).
The speciﬁcations of the datasets are summarized and the experimental results are reported in Table 3.
We can see that the proposed method outperforms others in most cases.
The closer π p is to 12 , the better classiﬁcation performance of BER and BER-FC.
In particular, in the experiments of MNIST and USPS ( π p = 0 . 5 ), BER and BER-FC can achieve comparable results as UU where pSVM falls behind.
This is because the balanced error assumption holds in this case.
However, in the experiments of pendigits and USPS ( π p = 0 . 1 , 0 . 9 ), our method still works well while the performance of
BER and BER-FC drops severely and become inferior to pSVM.
C ONCLUSIONS
We focused on training arbitrary binary classiﬁer, including deep networks, from only U data by
ERM.
We proved that this is impossible given a single set of U data, but it is possible given two sets of U data with different class priors, where all class priors necessary for training are given.
This led to an unbiased risk estimator and subsequently we proposed the ﬁrst ERM-based learning method from two sets of U data.
Experiments demonstrated that the proposed method could even successfully train AllConvNet and ResNet, and it compared favorably with state-of-the-art methods for learning from two sets of U data. [Table]
A P ROOFS
In this appendix, we prove all theorems.
A. P ROOF OF T HEOREM
We prove the theorem by contradiction, namely for any such p ( x, y ) , for all θ and a, b , we are able to ﬁnd some g for which (6) fails.
Our argument goes from the special (cid:96) 01 to the general (cid:96) in (7).
Firstly, let g ( x ) =
+ ∞ identically, so that (cid:96) ( g ( x ))
=
and (cid:96) ( − g ( x )) = 1 .
Plugging them into (3) and (6), we obtain that b = 1 − π p .
Secondly, let g ( x ) =
−∞ ; this time (cid:96) ( g ( x ))
= 1 and (cid:96) ( − g ( x ))
= 0 , and we obtain that a = π p .
Thirdly, let g ( x ) =
+ ∞ over p p and g ( x ) =
−∞ over p n , which is possible because g is arbitrary.
Hence, (cid:96) ( g ( x ))
=
and (cid:96) ( − g ( x ))
= 1 over p p as well as (cid:96) ( g ( x ))
= 1 and (cid:96) ( − g ( x ))
=
over p n , resulting in 0 = θb + (1 − θ ) a by solving which we know that θ =
π p / (2 π p − 1) .
This is a contradiction unless π p = 0 or π p = 1 , because 0 ≤ θ ≤whereas either π p / (2 π p − 1) < 0 or π p / (2 π p − 1) > 1 .
Finally, given any (cid:96) satisfying (7), it is not difﬁcult to verify that the three g above lead to the same contradiction with b = 1 − π p and the same a and θ .
A. P ROOF OF T HEOREM
By Deﬁnition 3, the learning objective is deﬁned by
[Equation]
then
[Equation]
Recall that the risk of g is deﬁned as (3):
[Equation]
In order to minimize (3)
[Equation]
it sufﬁces to minimize (14)
[Equation]
if we can make
[Equation]
Solving these equations gives us
[Equation]
which concludes the proof.
A. P ROOF OF T HEOREM
First, we show the uniform deviation bound, which is useful to derive the estimation error bound.
Lemma 6. For any δ > 0 , let C δ = (cid:112) (ln 2 /δ ) / 2 , then we have with probability at least 1 − δ ,
[Equation]
[Equation]
where the probability is over repeated sampling of data for evaluating (cid:98) R uu ( g ) . Proof.
Consider the one-side uniform deviation sup g ∈G (cid:98) R uu ( g ) − R ( g ) .
Since 0 ≤ (cid:96) ( z ) ≤ C (cid:96) , the change of it will be no more than C (cid:96) α/n if some x i is replaced, or no more than C (cid:96) α (cid:48) /n (cid:48)
if some x (cid:48)
j is replaced.
Subsequently, McDiarmid’s inequality (McDiarmid, 1989) tells us that
Pr
[Equation]
or equivalently, with probability at least 1 − δ/ 2 ,
[Equation]
By symmetrization (Vapnik, 1998), it is a routine work to show that
[Equation]
and according to Talagrand’s contraction lemma (Shalev-Shwartz & Ben-David, 2014),
[Equation]
The one-side uniform deviation sup g ∈G R ( g ) − (cid:98) R uu ( g ) can be bounded similarly.
Based on Lemma 6, the estimation error bound (13) is proven through
[Equation]
where (cid:98) R uu ( (cid:98) g uu ) ≤ (cid:98) R uu ( g ∗ ) by the deﬁnition of (cid:98) g uu .
B S YNTHETIC E XPERIMENTS WITH L INEAR M ODELS
Here we illustrate the problem and our method with a synthetic Gaussian mixture dataset.
We used two-dimensional Gaussian distributions with means µ + = [1 , 1] T and µ − =
[ − 1 , − 1] T and identity covariance for P and N class-conditional densities respectively.
Two unlabeled training datasets were drawn from Eq.
(1), where the training class priors are ( θ, θ (cid:48) )
= (0 . 9 , 0 . 4) and the sample sizes are ( n, n (cid:48) )
= (2000 , 1000) .
The test data were drawn from the original joint distribution p ( x, y ) for evaluating the performance of trained models and the test class prior is π p = 0 . 3 .
Note that the marginal distribution p ( x ) changes between training and testing stages (see Figure 1 (c), (d) and the right panel), which is the key difference between the problem of interest and the problem of learning from class-conditional label noise.
For the toy experiments, linear-in-input model g ( X ) =
ω T X
+ b , where ω ∈ R 2 and b ∈ R , and sigmoid loss were commonly used.
SGD with the learning rate 0. and mini-batch size 128 was employed for optimization.
For the purpose of clear comparison of the risk estimators, we did not add regularization in the toy experiments.
The learned models trained with 500 epochs are shown in Figure 1.
C B ENCHMARK EXPERIMENTS WITH NEURAL NETWORK MODELS C. E XPERIMENTAL S ETUP
MNIST The MNIST dataset contains 60,000 gray-scale training images and 10,000 test images (input dimension is 784) from handwritten digits 0 to 9.
Since the dataset has ten classes (0 to 9) originally, we used the even and odd digits to constitute the P and N classes respectively in order to construct a dataset for binary classiﬁcation, and the resulting class prior is 0 . 49 .
The model used for
MNIST was a ﬁve-layer fully connected neural network with ReLU as the activation functions: d300 - 300 - 300 - 300 - 1 .
Batch normalization was applied before hidden layers, and an (cid:96) 2 -regularization (the (cid:96) 2 coefﬁcient is ﬁxed to 0.0001) was also added.
The model was trained by SGD with an initial learning rate 0.001 and mini-batch size 128, and the learning rate was decreased according to the 1 / (1 + decay ∗ epochs ) schedule built in Keras, where the decay candidates were chosen from { 0 , 10 − 6 , 10 − 5 , 5 × 10 − 5 , 10 − 4 , 5 × 10 − 4 } .
Fashion-MNIST Fashion-MNIST is also a grayscale image dataset consisting of a training set of 60,000 examples and a test set of 10,000 examples (input dimension is 784), associated with a label from 10 fashion item classes.
We artiﬁcially corrupted it into a binary classiﬁcation dataset as follows: P class is formed by ‘T-shirt/top’, ‘Pullover’, ‘Coat’, ‘Shirt’, ‘Bag’, and N class is formed by ‘Trouser’, ‘Dress’, ‘Sandal’, ‘Sneaker’, ‘Ankle boot’, and the resulting class prior is 0 .
Moreover, the models and optimization of Fashion-MNIST follow the setup of MNIST, except that the initial learning rate was set to be 0.0001.
SVHN The SVHN dataset is a 32 × 32 color house-number image dataset containing 73,257 training images, 26,032 test images, and 531,131 extra training images.
We sampled 100,000 training examples from the concatenation of the training data and extra training data.
The extra training data were used in order to ensure enough training data to perform class prior change in Figure 2.
For SVHN, ‘0’, ‘6’, ‘8’, ‘9’ make up the P class, and ‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘7’ make up the N class, and the class prior is 0.27.
The model for SVHN was a 12-layer CNN: d - [ C (3 ∗ 3 , 96)] ∗ 2C (3 ∗ 3 , 96 , 2) -
[ C (3 ∗ 3 , 192)] ∗ 2 - C (3 ∗ 3 , 192 , 2) -C(
*3,192)- C (1 ∗ 1 , 192) - C (1 ∗ 1 , 10) - 1000 - 1000 - 1 , where C(3*3,96) means 96 channels of 3 × 3 convolutions followed by ReLU, [ · ]*2 means there are 2 such layers, and C(3*3,96,2) means a similar layer but with stride 2, etc., and the padding scheme is same padding.
Batch normalization was again applied and (cid:96) 2 -regularization (the (cid:96) 2 coefﬁcient is ﬁxed to 0.005) was also added.
We used Adam optimizer with the default momentum parameters β 1 = 0 .
9 and β 2 = 0 .
999 and learning rate 0.00001 and mini-batche size 500 to train the network.
CIFAR-
The CIFAR-10 dataset consists of 32 × 32 natural RGB images from 10 classes, with 5,000 training examples and 1,000 test examples in each class.
For CIFAR-10: P class is composed of ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’ and ‘horse’ with class prior 0 .
60 , and N class is composed of ‘airplane’, ‘automobile’, ‘ship’ and ‘truck’.
In CIFAR-10 experiments, we used the following 32layer ResNet architecture: d - C (3 ∗ 3 ∗ 16) -
[ C (3 ∗ 3 ∗ 16) , C (3 ∗ 3 ∗ 16)] ∗ 5 - [ C (3 ∗ 3 ∗ 32 , 2) , C (3 ∗ 3 ∗ 32)] - [ C ( ∗ 3 ∗ 32) , C (3 ∗ 3 ∗ 32)] ∗ 4 - [ C (3 ∗ 3 ∗ 64 , 2) , C (3 ∗ 3 ∗ 64)] -
[ C (3 ∗ 3 ∗ 64) , C (3 ∗ 3 ∗ 64)] ∗ 4 -Global average pool- 1 , where C ( ∗ 3 ∗ 16) means 16 channels of 3 × 3 convolutions followed by ReLU, [ · , · ] means a building block (He et al., 2016) for ResNet, [ · ] ∗ 5 means there are 5 such layers, C ( ∗ 3 ∗ 32 , 2) means similar layer but with stride 2, etc., and the padding scheme is same padding.
The optimization setup was the same as SVHN.
Training details For fair comparison, we used same network structures and optimization methods for the proposed method and the corresponding baselines.
In the experiments of Figure 2, we used all the training data for the 4 datasets and each U has half the amount of training data.
In the additional experiments of analysis of moving θ and θ (cid:48) closer and robustness of noisy training class priors, we need to sample more data from p p ( x ) (e.g. in the case of ( θ, θ (cid:48) )
= (0 . 9 , 0 . 4) ).
In order to ensure enough P data for sampling, we used 40,000 training data for MNIST and each U has 20, unlabeled training data.
C. S UPPLEMENTARY EXPERIMENTAL RESULTS
Due to limited space, here we report the means and standard deviations of the ﬁnal test risk based on the same 10 random samplings of benchmark experiments with deep neural networks (see Figure 2) in Table 4.
Note that small PN and PN oracle use fully labeled data generated from the same distribution as the test data, which are extremely advantageous.
The results show that the proposed
UU method outperforms small PN in most cases and is much better than small PN prior-shift.
Comparison of different losses We illustrate the operation of proposed unbiased risk estimator Eq.
(10) with logistic loss (cid:96) log ( z ) =
log (1 + exp( − z )) and its simpliﬁed version Eq.
(12) with sigmoid loss (cid:96) sig .
We tested on MNIST with ( θ, θ (cid:48) )
∈ { (0 . 9 , 0 . 1) , (0 . 8 , 0 . 2) , (0 . 7 , 0 . 3) } .
The experimental results are reported in Figure 4, where means and standard deviations of test risks based on the same 10 random samplings are shown.
We can see that the risk estimator Eq.
(10) with (cid:96) log performs similarly to Eq.
(12) with (cid:96) sig .
It is also interesting to see that the proposed UU classiﬁers are comparable to the PN oracle that can access fully supervised training data up to ( θ, θ (cid:48) )
= (0 . 8 , 0 . 2) .
For the harder case of ( θ, θ (cid:48) )
= (0 . 7 , [Table] [Figure]
Analysis of changing sample sizes of two U sets We also investigated the issue of covariate shift by changing the sample sizes of two U sets.
We ﬁxed n (cid:48) to 20000 and gradually moved n fromto 20000 , and tested the performance of UU and CCN on two different pairs of training class priors: { (0 . 9 , 0 . 4) , (0 . 8 , 0 . 4) } .
Each experiment was conducted 10 times and we reported the means and standard deviations of test risks in Figure 5.
The results show that as n moves farther from n (cid:48) , the proposed unbiased UU method works reasonably well, while the performance CCN drops severely.
It is because the larger covariate shift when n moves farther from n (cid:48) , which makes the CCN-based risk estimator more biased.
Due to the unbiasedness of the proposed risk estimator, UU classiﬁers can handle the covariate shift reasonably well.
The experimental results are consistent with analysis of moving θ and θ (cid:48) closer in Sec. 5.1.
[Figure]