L EARNING FROM P OSITIVE AND U NLABELED D ATA WITH A S ELECTION B IAS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of learning a binary classiﬁer only from positive data and unlabeled data ( PU learning ).
Recent methods of PU learning commonly assume that the labeled positive data are identically distributed as the unlabeled positive data.
However, this assumption is unrealistic in many instances of PU learning because it fails to capture the existence of a selection bias in the labeling process.
When the data has a selection bias, it is difﬁcult to learn the Bayes optimal classiﬁer by conventional methods of PU learning.
In this paper, we propose a method to partially identify the classiﬁer.
The proposed algorithm learns a scoring function that preserves the order induced by the class posterior under mild assumptions, which can be used as a classiﬁer by setting an appropriate threshold.
Through experiments, we show that the method outperforms previous methods for PU learning on various real-world datasets.
I NTRODUCTION
We consider a situation that there are only positive and unlabeled data, and train a binary classiﬁer only from them ( PU learning ).
This problem arises in various practical situations, such as information retrieval and outlier detection (Elkan & Noto, 2008;
Ward et al., 2009; Scott & Blanchard, 2009;
Blanchard et al., 2010; Li et al., 2009; Nguyen et al., 2011).
One of the milestones of PU learning is Elkan & Noto (2008), who proposed a practically useful algorithm with theoretical analysis, and there is subsequent research called unbiased PU learning (du Plessis & Sugiyama, 2014; du Plessis et al., 2015) where an unbiased estimator of the classiﬁcation risk is minimized.
We focus on the case-control scenario (a.k.a.
the problem setting based on two samples of data ; Ward et al., 2009; Niu et al., 2016).
In this scenario, positive data are obtained separately from unlabeled data, and unlabeled data are sampled from the whole population.
As Elkan & Noto (2008) explained, we cannot identify a classiﬁer without an assumption on how positive data are labeled.
Therefore “ selected completely at random ” (SCAR) is traditionally assumed, i.e., the positive labeled data are identically distributed as the positive unlabeled data (Elkan & Noto, 2008; du Plessis et al., 2015).
The assumption of SCAR, however, is unrealistic in many instances of PU learning, e.g., a patient’s electronic health record (Bekker & Davis, 2018a) and a recommendation system (Marlin & Zemel, 2009; Schnabel et al., 2016).
In these cases, there is a selection bias (Heckman, 1979; Manski, 2008; Angrist & Pischke, 2008); the distribution of the positive data may differ between the labeled data and the unlabeled data.
Several recent related work has also proposed alternative assumptions (Bekker & Davis, 2018a;b).
However, in order to weaken SCAR, they impose other additional assumptions.
In this work, we consider a more natural assumption such that SCAR becomes its special case.
We assume that the order over { x } induced by the conditional probability of observation p ( o = +1 | x ) is the same as the one induced by the conditional class probability p ( y = +
| x ) .
We call this property the invariance of order .
In the real-world application, there are many situations with a selection bias which follows the invariance of order.
Among them, we list the following two examples.
Example 1: (Anomaly Detection)
The goal of anomaly detection is to ﬁnd anomaly data in an unlabeled dataset based on another dataset that consists only of anomaly data.
When the anomaly data is obtained, the more likely a datum is an anomaly, the more likely it is noticed and gets labeled.
Here, we consider transductive learning.
It means, only from this two datasets, we classify the unlabeled dataset.
Example 2: (Face Recognition)
The goal of this task is to identify a user from a set of face images based on some pictures of the user.
Users may tend to provide pictures which are more easily identiﬁable as themselves.
In this case, the positive data is the user’s face and the unlabeled data consists of all users including the user.
The user may provide pictures in which the face can be seen clearly, while in the unlabeled data there may be a lot of pictures in which the user’s face is covered by the hair.
Our problem setting is similar to the problem called learning from instance-dependent noisy label (Du & Cai, 2015; Bootkrajang, 2016).
In this problem setting, positive and negative data are available, but some labels of data ﬂip from positive to negative and negative to positive with the instance-dependent probability as a noise.
Besides, there is existing work putting assumption similar to the invariance order (Du & Cai, 2015; Bootkrajang, 2016).
We explain the difference between this work and our work just after Assumption
We name our problem setting PU learning with a Selection Bias (PUSB).
In this paper, we propose a novel framework to deal with this problem setting.
The experimental results show that our proposed method is appropriate for real-world applications compared to existing approaches for PU learning.
P ROBLEM S ETTING OF PU L EARNING WITH A S ELECTION B IAS
We consider a binary classiﬁcation problem to classify x ∈ X ⊂ R d into one of the two classes {− 1 , +1 } .
We assume that there exists a joint distribution p ( x , y, o ) , where y ∈ {− 1 , +1 } is the class label of x , and o ∈ { 0 , 1 } is the observation status of y (observed if o =
+
and unobserved if o = 0 ).
In other words, x is labeled if o = +1 , and it is unlabeled otherwise.
In PU learning, there are two distinguished sampling schemes called one sample and two samples of data (Niu et al., 2016).
They are also called the censoring scenario and case-control scenario , respectively (Elkan & Noto, 2008).
In the censoring scenario, a set of unlabeled data is sampled from the marginal density p ( x ) .
Then, if a data point x is positive, this positive data gets labeled with probability p
( o = +1 | x ) ; if x is negative, it is never labeled.
In the case-control scenario, a set of positive data is drawn from the positive conditional density p ( x | y
= +1) and a set of unlabeled data is drawn from p ( x ) .
As Niu et al.
(2016) stated, the case-control scenario is slightly more general than the censoring scenario setting.
It is because the censoring scenario assumes the access to samples generated by p ( x ) , p ( x | o = +1) , and p ( x | o = 0) , but the case-control scenario only assumes the access to samples generated by p ( x ) and p ( x | o = +1) .
Therefore we focus on the case-control scenario.
Suppose that we have a positive dataset { x i } ni =1 and an unlabeled dataset { x (cid:48) i } n
[Equation]
i [Equation]
We assume that negative data are never labeled.
In addition, we do not assume SCAR.
Therefore, p ( x | y = +1) may differ from p ( x | y
= +1 , o = +1) .
In the case they differ, we say that there is a selection bias .
The quantity π = p ( y = +1) is called the class-prior .
In our problem setting, we consider the case where the class-prior is known.
For example, in anomaly detection, the percentage of anomaly in the whole batch of products can be reported based on past experiences.
Although there are various methods for estimating the class-prior in the traditional framework of the case-control scenario (du Plessis et al., 2016; Ramaswamy et al., 2016; Jain et al., 2016; Kato et al., 2018), we cannot estimate the class-prior in our problem setting under a theoretical guarantee.
In Section 5, we show how misspeciﬁed class priors affect the performance of a classiﬁer.
As we explain later, even if we do not know the class-prior, we only have to change the last step of our algorithm.
Our goal is to obtain a classiﬁer h :
X → {− 1 , 1 } only from { x i } ni =1 , { x (cid:48) i } n (cid:48)
i =1 , and π , which is justiﬁable under some kind of evaluation measure.
2. I DENTIFICATION S TRATEGY
As stated by Elkan & Noto (2008), even if the class prior is given, we cannot estimate p ( y = +1 | x ) only from positive data and unlabeled data without any assumption in PU learning.
In the case-control scenario, a standard assumption is SCAR, i.e. p ( x | y
=
+1 , o = +1)
=
p ( x | y
= +1 , o = 0) , so that p ( y = +
| x ) can be estimated from the data in principle.
However, in many instances of PU learning, the SCAR assumption is unreasonable.
Therefore, we relax SCAR and accommodate a selection bias.
We can see how SCAR makes the class posterior identiﬁable in the following equation:
[Equation]
Under the assumption of SCAR, we can estimate each of p ( x | y = +1) and p ( x ) from samples, in principle.
However, without assuming SCAR, p ( x | y = +1) may differ from p ( x | y
= +1 , o = +1) , and p ( y = +
| x ) is not identiﬁable.
Therefore, instead of estimating p ( y = +
{+| x ) and estimation of the Bayes-optimal classiﬁer and p ( y = +
+}
| x ) , we consider extracting some useful information of p ( y = +
| x ) to learn a classiﬁer.
This kind of approach is known as “partial identiﬁcation” (Manski, 2008) in statistics and economics.
The idea is to extract some useful information of a function instead of attempting to identify the whole function.
Firstly, we introduce an assumption that is weaker than SCAR.
Assumption 1 (Invariance of Order) .
For any x i , x j ∈ X , we have
[Equation]
Although Assumption 1 does not allow one to construct an unbiased estimator of the risk functional, we try to partially identify p ( y = +
| x ) under this assumption.
Our problem setting can be regarded as a generalization of the traditional case-control scenario because SCAR is a special case of the invariance of order.
A similar assumption can be found in the literature of learning from instance-dependent noisy label (Du & Cai, 2015; Bootkrajang, 2016), which consider a probabilistic label ﬂipping that is proportional to p ( y = [-+
-]+1 | x ) .
[-p ( o = +
-]However, in order to apply methods of learning from noisy label to PU learning, we need to assume the censoring scenario and these methods are not applicable to our problem setting based on the case-control scenario.
The censoring scenario is a special case of learning from noisy label where only [-if p ( y = +
 | x i ) = p ( y = +
-]negative data is contaminated, i.e., some positive labels ﬂip to negative labels.
Thus, in the censoring scenario, unlabeled data can be regarded as negative-labeled data contaminated by positive data.
[-We try to partially identify p ( y = +
 | x ) under this assumption. Our problem setting can be regarded as a generalization of-]On the other hand, in the case-control scenario, unlabeled data is generated from the marginal distribution p(x), i.e., we cannot observe samples generated from p(x|o = 0).
{+Therefore, our problem setting, namely the case-control scenario with invariance of order, is different from the existing work of learning from instance-dependent noisy label. In addition, our method is also applicable to the censoring scenario when the invariance of order holds because the unlabeled data of the case-control scenario can be made from positive and unlabeled data of the censoring scenario. In Example 1, p ( y = +1 | x ) is the probability that a given input x is anomaly, while p ( o = +
 | x ) is the probability that a given input x gets labeled in the dataset.+}
In Example 2, a positively labeled data is an image x that is known to belong to a user.
Here, p ( o = +1 | x ) is the probability that the user provides the picture x as a training datum.
T HE S TRATEGY FOR P ARTIAL I DENTIFICATION AND C LASSIFICATION
As we discussed in Section 2.1, we cannot estimate p ( y = +1 |
x ) when there is a selection bias even if the class prior is given.
Our idea of partial identiﬁcation is based on the following theorem.
Let r ( x ) be
[Equation]
Theorem 1 (Order preserving property of the score function) .
Suppose that Assumption 1 holds.
Then, for any x i , x j ∈ X ,
[-p ( y = +
-][Equation]
holds.
[-if and only if p ( y = +
-]
[-| x i ) < p ( y = +
 | x j ) .-] A proof is provided in Appendix A. Even though we cannot estimate p ( y = +1 |· ) , Theoremimplies that we can still extract the total order in X induced by p ( y = +1 |· ) if we can estimate r ( · ) .
Therefore, we propose to estimate r and use it as a score function that captures the total order induced by p ( y = +1 |· ) .
After obtaining an estimator of r (denoted by ˆ r ), we set a threshold θ ∈ R and use h ( x ) = sign ( r ( x ) − θ ) as a classiﬁer.
There are various ways of determining a threshold.
For instance, we put labels from data with the highest density ratio under a constraint of the number of data to which we can put labels (Hido et al., 2011).
Here, we introduce one useful principle for choosing θ .
We consider using a threshold θ π deﬁned by the following equation,
[Equation]
The intuition behind the deﬁnition of θ π is that the proportion of the positive data in the test data points should not deviate so much from the true class-prior.
This intuition becomes clearer in Section 4.3.
In Section 4, we discuss detailed methods for estimating r and setting θ based on data.
Our approach is summarized in the form of pseudo code in Algorithm 1.
In the rest of this section, we theoretically justify θ π deﬁned in (2).
Property of θ π : Let us consider the case where a classiﬁer is given as h ( x ) = sign ( r ( x ) − θ ) .
Then four population quantities, true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) (Lipton et al., 2014), that depend on r ( · ) and θ is written as follows:
[Equation]
Then, the precision and the recall of a classiﬁer are expressed as precision = (cid:16)
TP TP + FP (cid:17) and recall = (cid:16)
TP TP + FN (cid:17) , respectively.
For θ π , we have the following result.
Theorem 2.
If we use θ = θ π , then precision = recall holds.
A proof is shown in Appendix B. A threshold which makes the precision and the recall the same is known as precision–recall breakeven point (BEP)
(Sammut & Webb, 2010).
BEP is originally used to evaluate a generic classiﬁcation model with a score function and a threshold.
In addition, we can interpret BEP as a point which balances a prediction result.
As explained by Powers (2015), a classiﬁer using BEP as a threshold put the same weight to the false positives and false negatives.
Knowing BEP is also useful for deciding on a threshold which put unbalanced weight on the precision and the recall because we can tell if we are weighing precision more or recall more.
If the threshold is higher than BEP, we can tell that we are using a classiﬁer that puts more weight on the precision, and vice versa.
By setting the threshold relative to BEP, we can choose which of precision and recall is prioritized.
A LGORITHM
Here, we propose two directions for estimating r ( x ) = p ( x | y
=+1 ,o =+1) p ( x ) under the IO assumption, namely minimizing a pseudo classiﬁcation risk and direct density ratio estimation .
We also discuss how to set θ based on the data.
A pseudo-code of our algorithm is shown in Algorithm 2.
4. E STIMATION OF r BY MINIMIZING THE PSEUDO CLASSIFICATION RISK
Firstly, we introduce the minimization of the pseudo classiﬁcation risk.
The idea is to minimize the classiﬁcation risk used in traditional PU learning (du Plessis et al., 2014; 2015) as if there is no selection bias.
Under a selection bias, we cannot construct unbiased risk function, but the minimizer can be substituted for the density ratio in (1).
Conventional PU risk formulation: Let (cid:96) : R × {± 1 } → R + be a loss function, where R + is the set of non-negative real values, and F be a set of measurable functions from X to [ (cid:15), 1 − (cid:15) ] , where (cid:15) is a small positive value.
This (cid:15) is introduced to make the following optimization problem well-deﬁned.
du Plessis et al.
(2015) showed that the classiﬁcation risk of f ∈ F in the traditional PU problem setting with SCAR can be expressed as
[Equation]
where E p and E u are the expectations over p ( x | y = +1) and p ( x ) , respectively.
When there is no selection bias, we can simply replace the expectations with the corresponding sample averages to obtain an unbiased estimator of the classiﬁcation risk.
The pseudo classiﬁcation risk:
In our problem setting, we only have samples from p ( x | y
= +1 , o = +1) and not from p ( x | y = +1) .
Therefore, we cannot use our sample to obtain an empirical version of R PU .
However, we still consider the pseudo classiﬁcation risk of f ∈ F :
[Equation]
where E bias p is the expectation over p ( x | y
= +1 , o = +1) .
We call this functional the pseudo classiﬁcation risk because it is not the true classiﬁcation risk.
An unbiased estimator for the pseudo classiﬁcation risk can be obtained by replacing the expectations with the corresponding sample averages even if there is a selection bias.
For the loss function, we use the logarithmic loss: (cid:96) ( f ( x ) , +1))
= − log( f ( x )) and (cid:96) ( f ( x ) , − 1) =
− log(1 − f ( x )) .
In this case, the pseudo classiﬁcation risk of f ∈ F becomes
[Equation]
Justiﬁcation for minimizing the pseudo classiﬁcation risk: For the pseudo classiﬁcation risk with the logarithmic loss function, the following theorem justiﬁes its use.
Let us denote a minimizer of (4) by f ∗ , that is,
[Equation]
For the minimizer of (4), we derive the following theorem.
Theorem 3.
f ∗ is given by
[Equation]
where D 1 = { x | πp ( x | y
=
1 , o = +1) ≥ (cid:15)p ( x ) } and D 2 = { x | πp ( x | y
= 1 , o = +1) ≤ (1 − (cid:15) )
p ( x ) } .
A proof is provided in Appendix C. Theorem 3 implies that the minimization of the empirical version of the pseudo classiﬁcation risk allows us to estimate r .
Empirical Estimation: When we have training samples, we can naively replace the expectations by the corresponding sample averages.
For a hypothesis set H , which is a subset of a set of measurable functions, let us deﬁne the following risk minimization problem,
[Equation]
where ˆ E bias p denotes the empirical mean using the positive dataset with a selection bias and ˆ E u denotes the empirical mean using the unlabeled dataset.
du Plessis et al.
(2015) showed that, under
SCAR, the empirical version of the risk becomes unbiased toward the classiﬁcation risk.
However, Kiryo et al.
(2017) pointed out that unbiased PU learning does not work with deep learning.
Minimizing an empirical risk of (3) with deep learning easily causes over-ﬁtting because the risk is not bounded from below by 0 .
In order to implement PU learning with deep learning, Kiryo et al.
(2017) proposed the following non-negative risk,
[Equation]
After obtaining ˆ f , we set ˆ r = 1 π ˆ f .
Input: p ( x | y = +1) , p ( x ) and the class-pror π .
Using p ( x | y
= +1)
and p ( x ) , calculate r ( x ) by minimization of either (4) or (7). Using r ( x ) , calculate θ π in (2). Using the density ratio r ( x ) and the threshold θ π , obtain a classiﬁer h ( x ) = sign ( r ( x ) − θ π ) . 4. E STIMATION OF r BY D IRECT D ENSITY R ATIO E STIMATION
For another approach, we consider estimating the density ratio r ( x ) =
p ( x | y
=+1 ,o =+1) p ( x ) directly.
In order to estimate the ratio, there are various methods.
For example, we can estimate the probability density functions of the numerator and the denominator.
However, as Vapnik’s principle, we should avoid solving more difﬁcult intermediate problems.
Sugiyama et al.
(2012) summarized various methods estimating the density ratio directly.
Among existing methods of density ratio estimation, we employ Least-squares importance ﬁtting (LSIF), which uses the squared loss for density-ratio function ﬁtting.
The reason for this choice is that there is an algorithm called unconstrained Least-Squares
Importance Fitting (uLSIF), which is based on LSIF and allows us to obtain the closed-form solution that can be computed just by solving a system of linear equations.
Thus, uLSIF is numerically stable when it is regularized properly.
Moreover, the leave-one-out cross-validation score for uLSIF can also be computed analytically, which signiﬁcantly improves the computational efﬁciency in model selection.
Here, we introduce the formulation of LSIF.
Let S be a class of non-negative measurable functions s : X → R + .
We consider minimizing the following squared error between s and r :
[Equation]
The ﬁrst term of the last equation does not affect the result of minimization and we can ignore the term, i.e., the density ratio is estimated through the following minimization problem:
[Equation]
Empirical Estimation: As mentioned above, to minimize the empirical version of (7), we introduce uLSIF (See Kanamori et al.
For a hypotheses set H , we obtain ˆ r by
[Equation]
4. E MPIRICALLY ESTIMATING θ π
We consider replacing the threshold deﬁned by (2) with a value which can be calculated only from samples.
In the case that we have access to the test inputs
[Equation]
we choose a ˆ θ π that satisﬁes the following equation,
[Equation]
Here, we used the knowledge of π , the class-prior.
This choice of ˆ θ π amounts to classifying top- π test data as positive after ranking the inputs by ˆ r .
In the case we do not have access to the test inputs, we use held-out training data instead of test data to estimate the threshold analogically to (9).
E XPERIMENTS
In this section, we report experimental results which were conducted using synthetic data and real-world datasets 1) .
We used six classiﬁcation datasets, MNIST , shuttle , pageblocks , usps ,
Algorithm 2 PUSB
Input: A positive dataset { x i } ni =1 , an unlabeled dataset { x (cid:48) i } n (cid:48) i
=1 , a test dataset { x te i } n te i =1 and the class-pror π .
Using the positive dataset and the unlabeled dataset, estimate r ( x ) by any of (5), (6) or (8) and obtain ˆ r ( x ) .
Using ˆ r ( x ) , estimate θ π by (9) and obtain ˆ θ π . Using an estimator ˆ r ( x ) and ˆ θ , obtain a classiﬁer h ( x ) = sign (ˆ r ( x ) − ˆ θ π ) . [Table] connect-4 and spambase , from UCI repository 2) , CIFAR-10 3) and a document dataset obtained from SwissProt (Boeckmann et al., 2003) 4) .
MNIST and CIFAR-10 have 10 and 10 classes originally, and we constructed the positive and negative datasets from them as follows:
MNIST was preprocessed in such a way that 0 , 2 , 4 , 6 , 8 constitute the positive class, while 1 , 3 , 5 , 7 , 9 constitute the negative class; for CIFAR-10 , the positive dataset is formed by ‘airplane’, ‘automobile’, ‘ship’ and ‘truck’, and the negative dataset is formed by ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’ and ‘horse’.
Except for the document dataset, we show the details of datasets in Table 1 and made positive data with a selection bias based on estimators of p ( y = +1 | x ) .
For six datasets of the UCI repository and the CIFAR-10 , we made positive datasets with a selection bias artiﬁcially, but, for the document dataset, we have an unlabeled dataset and a positive dataset, which is gathered for classifying the labels in the unlabeled dataset.
We call unbiased PU learning proposed by du Plessis et al. (2015) “PU”, unbiased PU learning with a threshold estimated by (9) “PUSB”, uLSIF with a threshold estimated by (9) “DRSB”,
nonnegative
PU learning “nnPU” and nonnegative PU learning with a threshold estimated by (9) “nnPUSB”.
Let us denote a real-valued function as g ( x ) and use it for approximating f and s .
In the risk minimization of PU learning (4) and the density ratio estimation (7), we assume a generic real-valued function for g ( x ) , but we use linear-in-parameter model in the empirical risk of unbiased PU learning (5) and the density ratio estimation (8), and deep neural network in that of nonnegative PU learning (6).
When we used linear-in-parameter model, we assumed the following model,
[Equation]
where ϕ ( x ) =
[1 , ϕ 1 ( x ) , ..., ϕ m ( x )] (cid:62) is a set of basis functions.
For basis functions, we used the Gaussian functions centered around sample points ϕ (cid:96) ( x ) = exp (cid:0) −(cid:107) x − c (cid:96) (cid:107) 2 / (2 σ 2 ) (cid:1) , where { c 1 , ..., c m } = { x 1 , ..., x n , x (cid:48) 1 , ..., x (cid:48) n (cid:48) } and m = n + n (cid:48) .
A regularization parameter is given by λ and insert a regularization term λ 2 β (cid:62) β to the empirical risk of PU learning (5) and λ (cid:80) mi =1 β i to that of the density ratio estimation (8).
The model for deep neural network were explained in the following sections for each dataset.
We mainly used the same structure proposed in Kiryo et al.
(2017) in order to compare the performances.
For approximating s ( x ) , we used g ( x ) directly.
For approximating f , we used the sigmoid function:
[Equation]
The UCI data were downloaded from https://archive.ics.uci.edu/ml/index.php and https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ .
3) See https://www.cs.toronto.edu/~kriz/cifar.html .
The data can be downloaded from http://www.cs.ucsd.edu/users/elkan/posonly .
In this case, the loss is the same as the logistic loss and unbiased PU learning becomes convex.
In order to made a selection biases for the datasets, we estimated p ( y = +1 | x ) ﬁrstly.
For the estimation, we used all samples with the true positive and negative labels and logistic regression with the same model as that we used for PU learning.
5. T EST WITH S YNTHETIC D ATA
This experiment shows a classiﬁer given by PUSB.
We used samples from a mixture distribution of the following two class-conditional distributions:
[Equation]
where N ( µ, σ 2 ) denotes the univariate normal distribution with mean µ and variance σ 2 .
A positive dataset with a selection bias is sampled from
[Equation]
We generated 1000 positive samples and 10000 unlabeled samples.
We made two datasets with the different class-priors π = 0 .
3 , π = 0 .
We plotted three classiﬁers estimated by PU, PUSB and p ( y = +
| x ) .
The results are shown in Figure 1.
The classiﬁers obtained by our algorithm nearer to the Bayesian optimal classiﬁer for the classiﬁcation error rate, sign ( p ( y = +
| x ) − 1 / 2) .
5. B ENCHMARK T EST
In this subsection, we investigated the experimental performance in detail.
Linear-in-parameter model:
We used the mushrooms , shuttle , pageblocks , usps , connect-4 and spambase datasets.
Firstly, we estimated p ( y = +1 | x ) using the logistic regression with the same linear model.
Then, from positive data, we sampled positive dataset with a selection bias as
[Equation]
Then, we trained a classiﬁer by minimizing the empirical risk of PU learning (4) and the density ratio estimation ( 7 ).
For each binary labeled dataset, we made 12 different pairs of positive and unlabeled data with 4 different class-priors, { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } , and 3 different numbers of unlabeled data, { 800 , 1600 , 3200 } .
The number of positive data was ﬁxed at 400 .
We used 1000 test data sampled from the same distribution as the unlabeled data.
We ran the experiments 100 times and calculated the mean and standard deviation for the test dataset with PU, PUSB, and DRSB.
The results are shown in Table 2.
The classiﬁers obtained by our algorithm always show preferable performance to existing methods.
[Table]
error rate and the of the lower the and
Neural network model:
We used the MNIST and CIFAR-10 datasets.
The model of g ( x ) for the MNIST dataset was a 3 -layer multilayer perceptron (MLP) with ReLU (Nair & Hinton, 2010).
The model for the CIFAR-10 dataset was an all convolutional net (Springenberg et al., 2015).
Details of the network structure are shown in Appendix D.
Firstly, we estimated p ( y = +1 | x ) using the logistic regression with the same network structure and the positive and unlabeled datasets in the unlabeled dataset.
Next, from the positive dataset, we resampled positive dataset with an observation, which follows
[Equation]
Then, we trained a classiﬁer by minimizing (6) with the model deﬁned above.
We used 10000 test data sampled from the same distribution as the unlabeled data.
We ran the experiments 100 times and calculated the mean of the error rate, standard deviation of the error rate, the mean of recall and the mean of precision for each epoch in training with nnPU and nnPUSB.
The results are shown on the left side and center of Figure 2.
In the upper row, we show the mean and standard deviation of the error rate.
In the lower row, we show the mean of recall and the mean of precision.
As shown in Figure 2, the mean of the error rate of our algorithm was lower and the variance was also lower than the existing method.
As a matter of fact should be noted, for our results, FP =
FN held as we discussed in Section 4.3.
5. R EAL - WORLD D ATA T EST
We demonstrate the effectiveness of our algorithm in a real-world application, we used a document dataset recorded from the SwissProt database and released by Elkan & Noto (2008).
The dataset consists of three datasets, a positive dataset P, another positive dataset Q, and a negative dataset N.
We call this real dataset RealData .
The positive dataset P contains 2453 examples obtained from a specialized database named TCDB (Saier Jr et al., 2015).
Meanwhile, there are 4906 records in the unlabeled dataset U, which are randomly selected from SwissProt excluding those in the TCDB.
In other words, those 2 datasets P and U are disjoint.
Hence, this belongs to TS.
Furthermore, Das et al.
(2007) also manually labeled the unlabeled dataset.
They identiﬁed 348 positive members from the whole dataset {+N, i.e., the class prior is π = 348 / 
 = 0 . They+} called the new positive examples as Q and the leftover was N. We used Bag-of-Words to represent all examples of documents as vectors.
We represented each document as a 78894 -dimensional real-valued vector.
This real-world dataset is often used to evaluate algorithms of PU learning such as Elkan & Noto (2008);
He et al.
The model of g ( x ) for this dataset was a 5 -layer multilayer perceptron (MLP) with ReLU.
Details of the network structure are shown in Appendix D. We trained a classiﬁer using positive and unlabeled data.
Then, after ﬁnding a threshold estimated by (9), we evaluated the same evaluation measures as the previous experiments of a neural network by classifying U, i.e., we regarded the unlabeled data as test data.
The results are shown on the right side of Figure 2.
The result obtained from our algorithm outperformed the existing method.
In addition, we made positive data with a selection bias by ourselves for Benchmark Test, but FP=
FN also held in the real dataset with a selection bias.
It means that we cannot ignore a selection bias in real-world applications and our algorithm could be more useful in practice. 5. S ENSITIVITY A NALYSIS
In order to empirically measure the sensitivity of our algorithm to the class prior, we conduct two experiments using the RealData dataset in Section 5.3, whose class prior is estimated to be 0 . 0709 .
In the ﬁrst experiment, we trained a classiﬁer under misspeciﬁed class priors, namely 0 . 0209(= 0 .
0709 − 0 . 0500) , 0 . 1209(= 0709+0 . 0500) , 0 . 1709(= 0709+0 . 1000) , and 0 . 2709(= 0 . 0709 + 0 . 2000) .
In the second experiment, we empirically tested our algorithm with an estimator of the class prior by Ramaswamy et al.
(2016), which is considered to be the state-of-the-art method in the case-control scenario under SCAR.
Among the two methods proposed by Ramaswamy et al.
(2016), we adopted “KM2” for this experiment because “KM2” is reported to be better.
The estimated class prior was 0 .
All results are shown in Appendix E.
In so far as this dataset, our method works well under perturbations of the class prior or with an estimated class prior.
C ONCLUSION
In this paper, we proposed a novel framework for PU learning with a selection bias in positive data.
We put the IO assumption and showed the density ratio of labeled positive data and unlabeled data has the same order as the conditional class distribution for inputs.
Based on this result, we proposed an idea of partial identiﬁcation in which we ﬁrst estimate the density ratio and then use it as a classiﬁer by setting a threshold.
We conducted experiments to conﬁrm the effectiveness of our approach.
As we showed in the experiments, our method outperforms previous PU methods on real-world data.
A P ROOF OF T HEOREM
Proof.
We assumed that no negative data can be labeled.
As a result we have p ( o = {++
+} | x y = − 1) = 0 for arbitrary x ∈ X .
Therefore, p ( o = +1 | x )
=
p ( y = +
| x[-, o = +1) p ( x | o = +1) p ( y = +
 | o = +1) p ( x-] ) p ( o = +
| x , y = +1) + p ( y = [-+
-]− 1 | x ) p ( o = {++
 |+} x , y = − 1) =
p ( y = +
| x ) p ( o = +
| x , y = +1) .
By Bayes’ theorem, we can expand the density ratio in (1) as follows:
[-o = +
 | x ) p ( y = +
 | o = +1) p ( o = +1)-][Equation]
) p
[Equation] x )
Because p ( y =+1 | x ) p ( o =+1 | x ,y =+1) p ( o =+1 | x ) = 1 from (11), this is equivalent to r ( x )
=
Cp ( o = +
| x ) where
C = 1 p ( y =+1 ,o =+1) .
Hence, if Assumption 1 holds, for any x i , x j ∈ X {+, p ( y = +
+}
| x i ) ≤ p ( y = +
| x j ) ⇔ r ( x i ) ≤ r ( x j ) .
[-if and only if p ( y = +
 | x i ) = p ( y = +
 | x j )-] holds.
B P ROOF OF T HEOREM
Proof.
Regarding the probability as a classiﬁer, let us decide a threshold for p ( y = +
| x ) as follows:
[Equation]
This is equivalent to
[Equation]
The left hand side is equal to (cid:90)
[Equation]
| p
[Equation]
The right hand side is equal to
[Equation]
Hence, (12) is equivalent to the following equation,
[Equation]
From the deﬁnition of true positive (TP), false positive (FP), true negative (TN) and false negative (FN), the left hand side of the above equation is equal to
FP TP + FP + TN + FN and the right hand side of the above equation is equal to
FN TP + FP + TN + FN .
Therefore, this means that the threshold classiﬁes data as the number of negative data predicted as positive is equal to the number of positive data predicted as negative, i.e., FP = FN .
Then, we state the following lemma about the relationship between γ and θ .
Lemma 1.
For all x ∈ X , ( r ( x ) − θ )( p ( y = +
| x ) − γ ) ≥ 0 holds almost surely.
Proof.
By (2) and (12), the following equation also hold,
[Equation]
Hence, we can derive
[Equation]
This is equivalent to
[Equation]
From this equation, ( r ( x ) − θ )( p ( y = +
| x ) − γ ) ≥ 0 holds almost surely.
Therefore, if p ( y = +1 | x ) < γ , then r ( x ) < θ ; if p ( y = +1 | x ) ≥ γ , then r ( x ) ≥ θ almost surely.
Because precision = (cid:16)
[Equation] (cid:17) and recall = (cid:16)
[Equation]
that precision = recall .Therefore, using a threshold deﬁned by (2), we can obtain a classiﬁer whose precision and the recall are the same.
C P ROOF OF T HEOREM
Proof.
We minimize the functional R biasPU in the space of all functions on X taking values in [ (cid:15), 1 − (cid:15) ] , instead of F , and we will see that the minimizer matches the f ∗ stated in Theorem 3 which belongs to F .
Because we are optimizing
[Equation]
over all functions taking values in [ (cid:15), 1 − (cid:15) ] , the optimization is reduced to a point-wise minimization of l ( f ( x ))
[Equation]
=
[Equation]
[Equation]
( x
[Equation]
− log
[Equation]
1 − f ( x )))
− log(1 − f ( x ))
p ( x )) by considering f ( x ) as the optimization variable subject to f ( x ) ∈ [ (cid:15), 1 − (cid:15) ] .
The Karush-KuhnTucker (KKT) condition of this minimization problem is [Equation] where λ and µ are the Lagrange multipliers. The ﬁrst equation is equivalent to
[Equation]
From these conditions, the following results:
Assume λ = µ = 0 .
f ∗ ( x ) =
πp ( x | y
=+1) p ( x ) .
This solution satisﬁes the constraint only when x is in the domain D , where D = { x | p ( x | y = 1) ≤ (1 − (cid:15) )
p ( x ) } .
Assume λ > 0 and µ = 0 , f ∗ ( x ) = 1 − (cid:15) .
It is because inf f L ( f ; α, β ) → + ∞ as λ → + ∞ if f ∗ ( x ) < 1 − (cid:15) .
λ > 0 holds only when x is in the domain R \ D 2 , where R \ D 2 = { x | πp ( x | y = 1) <
(1 − (cid:15) )
p ( x ) } , because [Equation] [Equation] (cid:15)[-) 
-]
[Equation] ( p ( x ) + λ )(1 − (cid:15) [Equation]
λ (1 − (cid:15) ) [Equation]
+ λ (1 − (cid:15) )
+ p ( x )(1 − (cid:15) ) − πp ( x | y
= +1)
=⇔ λ = p ( x )(1 − (cid:15) ) − πp ( x | y = +1) (1 − (cid:15) )
+ (1 − (cid:15) ) .
Assume λ = 0 and µ > 0 , f ∗ ( x ) = (cid:15) .
It is because inf f L ( f ; α, β ) → + ∞ as µ → + ∞ if f ∗ ( x ) > (cid:15) .
This solution satisﬁes (13), only when x is in the domain R \ D 1 , where R \ D 1 = { x | πp ( x | y = 1) > (cid:15)p ( x ) } , because
[Equation]
Otherwise, there is no feasible solution.
As a result, the solution for the optimization problem f ∗ [Equation]
= arg min [Equation] [Equation] [Equation]
[Equation]
] R biasPU ( f ) is
[Equation]
( x
[Equation]
∈ D 2 ) , where D 1 = { x | πp ( x | y = 1 , o = +1) ≥ (cid:15)p ( x ) } and D 2 = { x | πp ( x | y = 1 , o = +1) ≤ (1 − (cid:15) ) p ( x ) } . Now, because the function f ∗ ( x ) constructed above is a measurable function if p ( x | y
=
1 , o = +1) and p ( x ) are measurable, the solution of the original optimization problem
[Equation] is equal to this f ∗ . D T HE N ETWORK S TRUCTURE USED IN S ECTION 5.2 AND S ECTION 5.  In Section 5.2, we used the MNIST and CIFAR-10 datasets.
The model of g ( x ) for the MNIST dataset was a 3 -layer multilayer perceptron (MLP) with ReLU (Nair & Hinton, 2010) (more speciﬁcally, 784 − 100 − 1 ). The model for the CIFAR-10 dataset was an all convolutional net (Springenberg et al., 2015): (32 × 32 × [ C (3 × 3 , 96)] ×
2- C (3 × 3 , 96 , 2)-[ C (3 × 3 , 192)]
×
2- C (3 × 3 , 192 , 2)- C (3 × 3 , 192)- C (1 × 1 , 192)- C (1 × 1 , 10)-1000-1000-1 , where the input is a 32 ×
RGB image, C (3 × 3 , 96) means 96 channels of 3 × 3 convolutions followed by ReLU, [ · ] × 2 means there are two such layers, C (3 × 3 , 96 , 2) means a similar layer but with stride 2, etc.; it is one of the best architectures for CIFAR-10 .
Batch normalization (Ioffe & Szegedy, 2015) was applied before hidden layers.
In Section 5.3, the model of g ( x ) for this dataset was a 5 -layer multilayer perceptron (MLP) with
ReLU (more speciﬁcally, 78894-300-300-300-300-1 ).
E E XPERIMENTAL R ESULTS OF S ECTION 5.
In order to empirically measure the sensitivity of a classiﬁer to the class prior, we conduct two experiments using the RealData dataset in Section 5.3, whose true class prior is 0 . 0709 .
In the ﬁrst experiment, we trained a classiﬁer under misspeciﬁed class priors, 0 . 0209(= 0 .
0709 − 0 . 0500) , 0 . 1209(=
0709+0 . 0500) , 0 . 1709(=
0709+0 . 1000) , and 0 . 2709(= 0 .
0709+0 . 2000) .
The results are shown in Figures 3–5.
In Figure 3, the test error in each misspeciﬁed class priors are shown.
In Figure 4, the precision and recall in each misspeciﬁed class prior is shown.
In both existing method and our method, misspeciﬁed class priors had a bad inﬂuence.
However, our method was less inﬂuenced by the misspeciﬁcation and showed better performance than the existing method.
Besides, the difference between the precision and recall of our algorithm is narrower than that of existing method as expected.
In order to discuss how misspeciﬁed class prior affects the precision and recall, we subtract the recall from the precision and show the values in Figure 5.
This result shows how the difference broadens as the class prior is misspeciﬁed worse.
In the second experiment, we empirically tested our algorithm with an estimated class prior.
In order to estimate the class prior, we used Ramaswamy et al.
(2016), which is considered to be the state-of-the-art method in the case-control scenario under SCAR.
Among two methods proposed by Ramaswamy et al.
(2016), we adopted “KM2” for this experiment because “KM2” is reported to be the better one.
The mean of the estimated class prior was 0 .
This experiment assumes the situation that we face in the real world.
The result is shown in Figure 5.
In so far as the
RealData dataset, our method still works empirically with an estimated class prior although there is no theoretical guarantee.
There is no theoretical justiﬁcation for using the existing methods of the class prior estimation in our problem setting.
Although we could not propose a universal approach for this problem, we still could apply the existing method to our problem setting for a speciﬁc dataset.