L EARNING FROM P OSITIVE AND U NLABELED D ATA WITH A S ELECTION B IAS
A BSTRACT
We consider the problem of learning a binary classiﬁer only from positive data and unlabeled data ( PU learning ).
Recent methods of PU learning commonly assume that the labeled positive data are identically distributed as the unlabeled positive data.
However, this assumption is unrealistic in many instances of PU learning because it fails to capture the existence of a selection bias in the labeling process.
When the data has selection bias, it is difﬁcult to learn the Bayes optimal classiﬁer by conventional methods of PU learning.
In this paper, we propose a method to partially identify the classiﬁer.
The proposed algorithm learns a scoring function that preserves the order induced by the class posterior under mild assumptions, which can be used as a classiﬁer by setting an appropriate threshold.
Through experiments, we show that the method outperforms previous methods for PU learning on various real-world datasets.
I NTRODUCTION
We consider a situation that there are only positive and unlabeled data, and train a binary classiﬁer only from them ( PU learning ).
This problem arises in various practical situations, such as information retrieval and outlier detection (Elkan & Noto, 2008;
Ward et al., 2009; Scott & Blanchard, 2009;
Blanchard et al., 2010; Li et al., 2009; Nguyen et al., 2011).
One of the milestones of PU learning is Elkan & Noto (2008), who proposed a practically useful algorithm with theoretical analysis, and there is subsequent research called unbiased PU learning (du Plessis & Sugiyama, 2014; du Plessis et al., 2015) where an unbiased estimator of the classiﬁcation risk is minimized.
We focus on the case-control scenario (a.k.a.
the problem setting based on two samples of data ; Ward et al., 2009; Niu et al., 2016).
In this scenario, positive data are obtained separately from unlabeled data, and unlabeled data are sampled from the whole population.
As Elkan & Noto (2008) explained, we cannot identify a classiﬁer without an assumption on how positive data are labeled.
Therefore “ selected completely at random ” (SCAR) is traditionally assumed, i.e., the positive labeled data are identically distributed as the positive unlabeled data (Elkan & Noto, 2008; du Plessis et al., 2015).
The assumption of SCAR is, however, unrealistic in many instances of PU learning, e.g., a patient’s electronic health record (Bekker & Davis, 2018a) and a recommendation system (Marlin & Zemel, 2009; Schnabel et al., 2016).
In these cases, there is a selection bias (Heckman, 1979; Manski, 2008;
Angrist & Pischke, 2008); the distribution of the positive data may differ between the labeled data and the unlabeled data.
Several recent related works have proposed alternative assumptions (Bekker & Davis, 2018a;b).
However, in order to weaken SCAR, they impose other additional assumptions.
In this work, we consider a more natural assumption such that SCAR becomes its special case.
We assume that p ( o = +1 | x {+, y = +1) and p ( y = +
 | x+} ) induce the same ordering on the input space X , where y ∈ {− 1 , +1 } is the data label and o = [-+
-]+1 (resp. o = 0 ) denotes the event that the data is observed (resp.
not observed).
We call this property the invariance of order .
In the real-world application, there are many situations with a selection bias which follows the invariance of order.
Among them, we list the following two examples.
Example 1: (Anomaly Detection)
The goal of anomaly detection is to ﬁnd anomaly data in an unlabeled dataset based on another dataset that consists only of anomaly data.
When the anomaly data is collected, the more likely a datum is an anomaly, the more likely it is noticed and gets labeled.
Example 2: (Face Recognition)
The goal of this task is to identify a user from a set of face images based on some pictures of the user.
In this case, the positive data are the face images identiﬁed with the user, and the unlabeled data consist of all the unidentiﬁed face images.
The user may more likely provide pictures in which the face can be seen clearly, while in the unlabeled data there may be many unclear images.
Our problem setting is similar to the problem called learning from instance-dependent noisy labels (Du & Cai, 2015; Bootkrajang, 2016).
In this problem setting, positive and negative data are available, but the labels are subject to the noise that ﬂips the label with instance-dependent probability.
Besides, there are existing works putting assumption similar to the invariance order (Du & Cai, 2015; Bootkrajang, 2016).
We explain the difference between their works and ours around Assumption 1 in
Section 2.1.
We name our problem setting PU learning with a Selection Bias (PUSB).
In this paper, we propose a novel framework to deal with this problem setting.
The experimental results show that our proposed method is appropriate for real-world applications compared to existing approaches for PU learning.
P ROBLEM S ETTING OF PU L EARNING WITH A S ELECTION B IAS
We consider a binary classiﬁcation problem to classify x ∈ X ⊂ R d into one of the two classes {− 1 , +1 } .
We assume that there exists a joint distribution p ( x , y, o ) , where y ∈ {− 1 , +1 } is the class label of x , and o ∈ { 0 , +1 } is the observation status of y (observed if o =
+
and unobserved if o = 0 ).
In other words, x is labeled if o = +1 , and it is unlabeled otherwise.
In PU learning, there are two distinguished sampling schemes called one sample and two samples of data (Niu et al., 2016).
They are also called the censoring scenario and case-control scenario , respectively (Elkan & Noto, 2008).
In the censoring scenario, a set of unlabeled data is sampled from the marginal density p ( x ) .
Then, if a data point x is positive, it gets labeled with probability p
( o = {++
+} | x , y = +1) ; if x is negative, it is never labeled.
In the case-control scenario, a set of positive data is drawn from the positive conditional density p ( x | y
= +1) and a set of unlabeled data is drawn from p ( x ) .
As Niu et al.
(2016) stated, the case-control scenario is slightly more general than the censoring scenario setting.
It is because the censoring scenario assumes the access to samples generated by p ( x ) , p ( x | o = +1) , and p ( x | o = 0) whereas the case-control scenario only assumes the access to samples generated by p ( x ) and p ( x | o = +1) .
Therefore we focus on the case-control scenario.
Suppose that we have a positive dataset { x i } ni =1 and an unlabeled dataset { x (cid:48) i
[Equation]
n (cid:48) i =
[Equation]
We assume that negative data are never labeled.
Note that we do not assume SCAR.
Therefore, p ( x | y = +1) may differ from p ( x | y
= +1 , o = +1) .
In the case they differ, we say that there is a selection bias .
The quantity π = p ( y = +1) is called the class-prior .
In our problem setting, we assume that π is known.
For example, in anomaly detection, the percentage of anomaly in the whole batch of products can be reported based on past experiences.
Although there are various methods for estimating the class-prior in the traditional framework of the case-control scenario (du Plessis et al., 2016; Ramaswamy et al., 2016; Jain et al., 2016; Kato et al., 2018), we cannot estimate the class-prior in our problem setting under a theoretical guarantee.
In Section 5, we show how misspeciﬁed class priors affect the performance of a classiﬁer.
As we explain later, even if we do not know the class-prior, we only have to change the last step of our algorithm.
In summary, our goal is to obtain a classiﬁer h :
X → {− 1 , 1 } only from { x i } n i =1 , { x (cid:48) i } n (cid:48)
i =1 , and π under a weaker assumption than SCAR.
2. I DENTIFICATION S TRATEGY
As stated by Elkan & Noto (2008), even if the class prior is given, we cannot estimate p ( y = +1 | x ) only from positive data and unlabeled data without any assumption in PU learning.
In the case-control scenario, a standard assumption is SCAR, i.e. p ( x | y
=
+1 , o = +1)
=
p ( x | y
= +1 , o = 0) , so that p ( y = +
| x ) can be estimated from the data in principle.
However, in many instances of PU learning, the SCAR assumption is unreasonable as discussed before.
Therefore, we relax SCAR and accommodate a selection bias.
We can see how SCAR makes the class posterior identiﬁable in the following equation:
[Equation]
SCAR [Equation] Here, p ( x | y = +1 , o = +1) can be estimated from the sample.
This estimate can be used to obtain an estimate of p ( x | y = +1) and hence that of p ( y = +1 | x ) if we assume SCAR. However, without assuming SCAR, p ( x | y = +1) may differ from p ( x | y
= +1 , o = +1) , and p ( y = +
| x ) is not identiﬁable.
Therefore, instead of estimating p ( y = +
[-| x ) and estimation of the Bayes-optimal classiﬁer and p ( y = +
-]
| x ) , we consider extracting some useful information of p ( y = +
| x ) to learn a classiﬁer.
This kind of approach is known as “partial identiﬁcation” (Manski, 2008) in statistics and economics.
First, we introduce an assumption that is weaker than SCAR.
Assumption 1 (Invariance of Order) .
For any x i , x j ∈ X , we have
[Equation]
Although Assumption 1 does not allow one to construct an unbiased estimator of the risk functional, we try to partially identify p ( y = +
| x ) under this assumption.
Our problem setting can be regarded as a generalization of the traditional case-control scenario because SCAR is a special case of the invariance of order. 2. R ELATED W ORKS
A similar assumption can be found in the literature of learning from instance-dependent noisy labels (Du & Cai, 2015; Bootkrajang, 2016), which considers a probabilistic label ﬂipping that is proportional to p ( y = +1 | x ) .
However, in order to apply methods of learning from noisy labels to PU learning, we need to assume the censoring scenario and these methods cannot be applied to our problem setting based on the case-control scenario.
The censoring scenario is a special case of learning from noisy labels where only negative data is contaminated, i.e., some positive labels ﬂip to negative labels.
Thus, in the censoring scenario, unlabeled data can be regarded as negative-labeled data contaminated by positive data.
On the other hand, in the case-control scenario, unlabeled data is generated from the marginal distribution p ( x ) , i.e., we cannot observe samples generated from p ( x | o = 0) .
Therefore, our problem setting, namely the case-control scenario with invariance of order, is different from the existing works of learning from instance-dependent noisy labels.
In addition, our method is also applicable to the censoring scenario when the invariance of order holds because the unlabeled data of the case-control scenario can be made from positive and unlabeled data of the censoring scenario.
In Example 1, p ( y = +1 | x ) is the probability that a given input x is anomaly, while p ( o = +
| x ) is the probability that a given input x gets labeled in the dataset.
In Example 2, a positively labeled data is an image x that is known to belong to a user.
Here, p ( o = +1 | x ) is the probability that the user provides the picture x as a training datum.
S TRATEGY FOR P ARTIAL I DENTIFICATION AND C LASSIFICATION
As discussed in Section 2.1, we cannot estimate p ( y = {++
+}
| x ) when there is a selection bias even if the class prior is given.
Our idea of partial identiﬁcation is based on the following theorem with the density ratio
[Equation]
Input: p ( x | y = +1) , p ( x ) and the class-pror π .
Using p ( x | y
= +1)
and p ( x ) , calculate r ( x ) by minimization of either (4) or (7).
Using r ( x ) , calculate θ π in (2).
Using the density ratio r ( x ) and the threshold θ π , obtain a classiﬁer h ( x ) = sign ( r ( x ) − θ π ) .
Theorem 1 (Order Preserving Property of the score Function) .
Suppose that Assumption 1 holds.
Then, for any x i , x j ∈ X ,
[Equation]
A proof is provided in Appendix A. Even though we cannot estimate p ( y = +1 |· ) , Theorem 1 implies that we can still extract the total order in X induced by p ( y = {++
+} |· ) if we can estimate r ( · ) .
Therefore, we propose to estimate r and use it as a score function that captures the total order induced by p ( y = +1 |· ) .
After obtaining an estimator of r (denoted by ˆ r ), we set a threshold θ ∈ R and use h ( x ) = sign ( r ( x ) − θ ) as a classiﬁer.
There are various ways of determining the threshold.
For instance, we put labels from data with the highest density ratio under a constraint on the number of data to which we can put labels (Hido et al., 2011).
Here, we introduce one useful principle for choosing θ .
We consider using a threshold θ π deﬁned by the following equation,
[Equation]
The intuition behind the deﬁnition of θ π is that the proportion of the positive data in the test data points should not deviate so much from the class-prior.
This intuition becomes clearer in Section 4.3.
In Section 4, we discuss detailed methods for estimating r and setting θ based on data.
Our approach is summarized in the form of a pseudo-code in Algorithm 1.
In the rest of this section, we theoretically justify θ π deﬁned in (2).
Property of θ π : Let us consider the case where a classiﬁer is given as h ( x ) = sign ( r ( x ) − θ ) .
Then four population quantities, true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) (Lipton et al., 2014), that depend on r ( · ) and θ is written as follows:
[Equation]
Then, the precision and the recall of a classiﬁer are expressed as precision = (cid:16)
TP TP + FP (cid:17) and recall = (cid:16)
TP TP + FN (cid:17) , respectively.
For θ π , we have the following result.
Theorem 2.
If we use θ = θ π , then precision = recall holds.
A proof is shown in Appendix B. The threshold θ π is known as precision–recall breakeven point (BEP)
(Sammut & Webb, 2010), which makes the precision and the recall the same.
BEP is originally used to evaluate a generic classiﬁcation model with a score function and a threshold.
Besides, we can interpret BEP as a point which balances a prediction result; as explained by Powers (2015), a classiﬁer using BEP as a threshold puts the same cost to the false positives and false negatives.
Knowing BEP is also useful for deciding on a threshold which put unbalanced weight on the precision and the recall because we can tell if we are weighing precision more or recall more.
A LGORITHM
Here, we propose two directions for estimating r ( x ) = p ( x | y
=+1 ,o =+1) p ( x ) under the assumption of invariance of order, namely minimizing a pseudo classiﬁcation risk and direct density ratio estimation .
We discuss how to set θ based on the data.
A pseudo-code of our algorithm is shown in Algorithm 2.
Input: A positive dataset { x i } ni =1 , an unlabeled dataset { x (cid:48) i } n (cid:48) i
=1 , a test dataset { x te i } n te i =1 and the class-pror π .
Using { x i } ni =1 and { x (cid:48) i } n (cid:48)
i =1 , estimate r ( x ) by any of (5), (6) or (8) and obtain ˆ r ( x ) . Using ˆ r ( x ) , estimate θ π by (9) and obtain ˆ θ π . Using an estimator ˆ r ( x ) and ˆ θ , obtain a classiﬁer h ( x ) = sign (ˆ r ( x ) − ˆ θ π ) . 4. E STIMATION OF r BY MINIMIZING PSEUDO CLASSIFICATION RISK
First, we introduce the minimization of the pseudo classiﬁcation risk.
The idea is to minimize the classiﬁcation risk used in traditional PU learning (du Plessis et al., 2014; 2015) as if there is no selection bias.
Under a selection bias, we cannot construct unbiased risk function, but the minimizer can be substituted for the density ratio in (1).
Conventional PU risk formulation: Let (cid:96) : R × {± 1 } → R + be a loss function, where R + is the set of non-negative real values, and F be the set of measurable functions from X to [ (cid:15), 1 − (cid:15) ] , where (cid:15) ∈ (0 , 1 / 2) is a small positive value.
This constant (cid:15) is introduced to make the following optimization problem well-deﬁned.
du Plessis et al.
(2015) showed that the classiﬁcation risk of f ∈ F in the traditional PU problem setting with SCAR can be expressed as
[Equation]
where E p and E u are the expectations over p ( x | y = +1) and p ( x ) , respectively.
When there is no selection bias, we can replace the expectations with the corresponding sample averages to obtain an unbiased estimator of the classiﬁcation risk.
The pseudo classiﬁcation risk:
In our problem setting, we only have samples from p ( x | y
= +1 , o = +1) and not from p ( x | y = +1) .
Therefore, we cannot use our sample to obtain an empirical version of R PU .
However, we still consider the pseudo classiﬁcation risk of f ∈ F :
[Equation]
where E bias p is the expectation over p ( x | y
= +1 , o = +1) .
We call this functional the pseudo classiﬁcation risk because it is not the true classiﬁcation risk.
An unbiased estimator for the pseudo classiﬁcation risk can be obtained by replacing the expectations with the corresponding sample averages even if there is a selection bias.
For the loss function, we use the logarithmic loss: (cid:96) ( f ( x ) , +1))
= − log( f ( x )) and (cid:96) ( f ( x ) , − 1) =
− log(1 − f ( x )) .
In this case, the pseudo classiﬁcation risk of f ∈ F becomes
[Equation]
Justiﬁcation for minimizing the pseudo classiﬁcation risk: For the pseudo classiﬁcation risk with the logarithmic loss function, the following theorem justiﬁes its use.
Let us denote a minimizer of (4) by f ∗ , that is,
[Equation]
For the minimizer of (4), we show the following theorem.
Theorem 3.
It holds almost everywhere that
[Equation]
where D 1 = { x | πp ( x | y
=
1 , o = +1) ≥ (cid:15)p ( x ) } and D 2 = { x | πp ( x | y
= 1 , o = +1) ≤ (1 − (cid:15) )
p ( x ) } .
A proof is provided in Appendix C. Theorem 3 implies that the minimization of the empirical version of the pseudo classiﬁcation risk allows us to estimate r .
Empirical Estimation: When we train a classiﬁer with training samples, we can naively replace the expectations with the corresponding sample averages.
For a hypothesis set H , which is a set of measurable functions, let us deﬁne the following risk minimization problem,
[Equation]
where ˆ E bias p denotes the averaging operator over positive data with a selection bias, ˆ E u denotes the averaging over the unlabeled data, and R is a regularization term.
du Plessis et al.
(2015) showed that, under SCAR, the empirical version of the risk becomes unbiased toward the classiﬁcation risk.
However, Kiryo et al.
(2017) pointed out that unbiased PU learning does not work with deep neural networks.
Minimizing an empirical risk of (3) with deep neural networks easily causes over-ﬁtting because the risk is not bounded from below by 0 .
In order to implement PU learning with deep neural networks, Kiryo et al.
(2017) proposed the following non-negative risk,
[Equation]
where ( · ) + := max { 0 , ·} . After obtaining ˆ f , we construct an estimator of the density ratio r by ˆ r = {+
+}
π ˆ f 1 or ˆ r = 1 π ˆ f 2 .
4. E STIMATION OF r BY D IRECT D ENSITY R ATIO E STIMATION
For another approach, we consider estimating the density ratio r ( x ) =
p ( x | y
=+1 ,o =+1) [Equation] x ) directly.
We can estimate the probability density functions of the numerator and the denominator.
However, as known as Vapnik’s principle, we should avoid solving more difﬁcult intermediate problems than the target problem.
Sugiyama et al.
(2012) summarized methods estimating the density ratio directly.
Among existing methods, we employ Least-squares importance ﬁtting (LSIF), which uses the squared loss for density-ratio function ﬁtting.
The reason for this choice is that there is an algorithm called unconstrained Least-Squares Importance Fitting (uLSIF) with a computational advantage. We can obtain the closed-form solution just by solving the linear equations.
Thus, uLSIF is numerically stable when it is regularized properly.
Moreover, the leave-one-out cross-validation score for uLSIF can also be computed analytically, which signiﬁcantly improves the computational efﬁciency in model selection.
Here, we introduce the formulation of LSIF.
Let S be the class of non-negative measurable functions s : X → R + .
We consider minimizing the following squared error between s and r :
[Equation]
The ﬁrst term of the last equation does not affect the result of minimization and we can ignore the term, i.e., the density ratio is estimated through the following minimization problem:
[Equation]
Empirical Estimation: As mentioned above, to minimize the empirical version of (7), we use uLSIF (Kanamori et al., 2009).
Given a hypothesis class H , we obtain ˆ r by
[Equation]
where R is a regularization term.
4. E STIMATION OF θ π
We consider replacing the threshold deﬁned by (2) with a value which can be calculated only from samples.
By using the test inputs or held-out training data, { x te i } n te i =1 ∼ p ( x ) , we ﬁnd ˆ θ π that satisﬁes
[Table]
the following equation,
[Equation]
Here, we used the knowledge of π , the class-prior.
This choice of ˆ θ π amounts to classifying top- π test data as positive after ranking the inputs by ˆ r .
E XPERIMENTS
In this section, we report experimental results which were conducted using synthetic data and real-world datasets 1) .
We used six classiﬁcation datasets, MNIST , shuttle , pageblocks , usps , connect-4 and spambase , from UCI repository 2) , CIFAR-10 3) and a document dataset obtained from SwissProt (Boeckmann et al., 2003) 4) .
MNIST and CIFAR-10 originally have 10 andclasses, respectively, and we constructed the positive and negative datasets from them as follows:
MNIST was preprocessed in such a way that 0 , 2 , 4 , 6 , 8 constitute the positive class, while 1 , 3 , 5 , 7 , 9 constitute the negative class; for CIFAR-10 , the positive dataset is formed by ‘airplane’, ‘automobile’, ‘ship’ and ‘truck’, and the negative dataset is formed by ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’ and ‘horse’.
Except for the document dataset, we show the details of datasets in Table 1 and made positive data with a selection bias based on estimators of p ( y = {++
+} | x ) as we show in each experiments.
For six datasets of the UCI repository and the CIFAR-10 , we made positive datasets with a selection bias artiﬁcially, but, for the document dataset, we have an unlabeled dataset and a positive dataset, which is gathered for classifying the labels in the unlabeled dataset.
We call unbiased PU learning proposed by du Plessis et al.
(2015) “PU”, unbiased PU learning with a threshold estimated by (9) “PUSB”, uLSIF with a threshold estimated by (9) “DRSB”,
nonnegative
PU learning “nnPU” and nonnegative PU learning with a threshold estimated by (9) “nnPUSB”.
For the hypothesis class H in the density ratio estimation (8), we use the linear-in-parameter model:
[Equation]
where ϕ ( x ) =
[1 , ϕ 1 ( x ) , ..., ϕ m ( x )] (cid:62) is a vector of basis functions.
For basis functions, we used the Gaussian kernel located at sample points ϕ (cid:96) ( x ) = exp (cid:0) −(cid:107) x − c (cid:96) (cid:107) 2 / (2 σ 2 ) (cid:1) , where { c 1 , ..., c m } = { x 1 , ..., x n , x (cid:48) 1 , ..., x (cid:48) n (cid:48) } and m = n + n (cid:48) .
For the hypothesis class H in the risk minimization of PU learning (5), we use the following model with the sigmoid function: [Equation] In this case, the loss is the same as the logistic loss and unbiased PU learning becomes convex. For the hypothesis class H in the risk minimization of nonnegative PU learning (6), we use deep neural networks.
The speciﬁcations of deep neural networks are given in the following sections for each dataset.
We mainly used the same structure proposed in Kiryo et al.
(2017) in order to compare the performances.
For the regularization term R , we used the (cid:96) 2 norm of the parameters scaled by a positive scalar λ for all methods.
The source code is available at https://github.com/MasaKat0/PUlearning .
The UCI data were downloaded from https://archive.ics.uci.edu/ml/index.php and https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ .
3) See https://www.cs.toronto.edu/~kriz/cifar.html .
The data can be downloaded from http://www.cs.ucsd.edu/users/elkan/posonly .
5. T EST WITH S YNTHETIC D ATA
This experiment shows the classiﬁer given by PUSB.
We used samples from a mixture distribution of the following two class-conditional distributions:
[Equation]
where N ( µ, σ 2 ) denotes the univariate normal distribution with mean µ and variance σ 2 .
A positive dataset with a selection bias was sampled from p
[Equation]
We generated 1 , 000 positive samples and 10 , 000 unlabeled samples.
We made two datasets with different class-priors π = 0 .
3 and π = 0 .
[-We plotted three classiﬁers estimated by PU, PUSB and p ( y = +
 | x ) . The results are shown in-] Figure 1 shows classiﬁers constructed by PU and PUSB along with the Bayes optimal classiﬁer sign ( p ( y = +
| x ) − 1 / 2) .
The classiﬁer of PUSB is closer to the Bayes optimal classiﬁer than that of PU.
5. T EST WITH B ENCHMARK D ATA
Here, we investigate the experimental performance in detail.
Linear-in-parameter model:
We used the mushrooms , shuttle , pageblocks , usps , connect-4 and spambase datasets.
First, we estimated p ( y = +1 | x ) using the logistic regression with the same linear model.
Then, we obtained the labeled positive data by labeling some instances of the positive data following
[Equation]
Then, we trained a classiﬁer by minimizing the empirical risk of PU learning (4) and the density ratio estimation ( 7 ).
For each binary labeled dataset, we made 12 different pairs of positive and unlabeled data with 4 different class-priors, { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } , and 3 different numbers of unlabeled data, { 800 , 1600 , 3200 } .
The number of positive data was ﬁxed at 400 .
We used 1000 test data sampled from the same distribution as the unlabeled data.
We ran the experiments 100 times and calculated the mean and standard deviation for the test dataset with PU, PUSB, and DRSB.
The results are shown in Table 2.
The classiﬁers obtained by our algorithm always show preferable performance to existing methods.
Neural network model:
We used the MNIST and CIFAR-10 datasets.
For MNIST , a 3 -layer multilayer perceptron (MLP) with ReLU activation (Nair & Hinton, 2010) was used.
For CIFAR-10 , an all convolutional net (Springenberg et al., 2015) was used.
Details of the network structure are shown in Appendix D.
[Table]
are the error rates and the vertical axes of the bottom ﬁgures are the precision and the recall.
First, we estimated p ( y = +1 | x ) using the logistic regression with the same network structure using the positive and negative datasets in the unlabeled dataset.
Next, from the positive dataset, we resampled positive dataset with an observation, which follows
[Equation]
Then, we trained a classiﬁer by minimizing (6) with the model deﬁned above.
We used 10 , 000 test data sampled from the same distribution as the unlabeled data.
We ran the experiments 100 times and calculated the mean of the error rate, the standard deviation of the error rate, the mean of recall and the mean of precision for each epoch in training with nnPU and nnPUSB.
The results are shown on the left side and center of Figure 2.
In the upper row, we show the mean and standard deviation of the error rate.
In the lower row, we show the mean of recall and the mean of precision.
As shown in Figure 2, the mean of the error rate of our algorithm is lower and the variance is also lower than the existing method.
As we discussed in Section 4.3, FP = FN is also empirically observed.
5. T EST WITH R EAL - WORLD D ATA
In the previous sections, we artiﬁcially made positive data with a selection bias. Here we demonstrate the effectiveness of our algorithm in real-world data with a selection bias. We used a document dataset based on the SwissProt database released by Elkan & Noto (2008).
We call this dataset RealData .
The dataset originally contained 2,453 labeled positive examples (P) and 4,906 unlabeled examples (U).
The unlabeled examples were labeled later by Das et al.
As a result, the dataset is [-π = 348 / 
 = 0 . They called the new positive examples as Q and-]likely to have a natural observation bias in the P data while it allows an access to the ground-truth labels for all data. Out of the U data, 348 examples are positive and the rest are negative.
The class prior is π = 348 / 4 , 906 = 0 .
We used Bag-of-Words to represent the documents as 78 , 894 -dimensional vectors.
Details of the network structure are shown in Appendix D. We trained a classiﬁer using positive and unlabeled data.
Then, after ﬁnding a threshold estimated by (9), we evaluated the same evaluation measures as the previous experiments by classifying U, i.e., we regarded the unlabeled data as test data.
As shown on the right of Figure 2, the result of our algorithm outperforms the existing method.
5. T EST FOR U NKNOWN C LASS PRIOR
In order to evaluate how our algorithm works for the case that the class prior is unknown, we empirically tested our algorithm with an estimator of the class prior.
We used the RealData dataset in Section 5.3, whose class prior is 0 . 0709 .
For the class prior estimator, we used the KM2 method by Ramaswamy et al.
(2016), which is considered to be the state-of-the-art method in the case-control scenario under SCAR.
The estimated class prior was 0 .
1095 and the result is shown in Figure 3.
We can see from the ﬁgure that our method works well with an estimated class prior in so far as RealData dataset is concerned.
We also show results for the classiﬁers trained under misspeciﬁed class priors in Appendix E, which also shows that our method works stably for misspeciﬁed class priors.
C ONCLUSION
In this paper, we proposed a novel framework for PU learning with a selection bias in positive data.
We put the assumption of the invariance of order and showed the density ratio of labeled positive data and unlabeled data has the same order as the class-conditional distribution for inputs.
Based on this result, we proposed a method based on partial identiﬁcation in which we ﬁrst estimate the density ratio and then use it as a classiﬁer by setting a threshold.
We conducted experiments to conﬁrm the effectiveness of our approach.
As we showed in the experiments, our method outperforms previous PU methods on real-world data.
A CKNOWLEDGMENTS
This work was supported by JSPS KAKENHI 16H00881 and the AIP challenge program, Japan.
A P ROOF OF T HEOREM
Proof.
We assumed that no negative data can be labeled.
As a result we have p ( o = +
| x y = − 1) = 0 for arbitrary x ∈ X .
Therefore, p ( o = +1 | x )
=
p ( y = +
| x ) p ( o = +
| x , y = +1)
+ p ( y = − 1 | x ) p ( o = +
| x , y = − 1) =
p ( y = +
| x ) p ( o = +
| x , y = +1) .
By Bayes’ theorem, we can expand the density ratio in (1) as follows:
[Equation]
[Equation]
1) p ( x ) . Because p ( y =+1 | x ) p ( o =+1 | x ,y =+1) p ( o =+1 | x ) = 1 from (11), this is equivalent to r ( x )
=
Cp ( o = +
| x ) , where C = 1 p ( y =+1 ,o =+1) .
Hence, if Assumption 1 holds, for any x i , x j ∈ X , p ( y = +
| x i ) ≤ p ( y = +
| x j ) ⇔ r ( x i ) ≤ r ( x j ) .
B P ROOF OF T HEOREM
Before proving Theorem 2, we consider a threshold [-for p ( y = +
 | x )-]deﬁned as follows:
[Equation]
Then, we state the following lemma on the relationship between γ and θ π .
Lemma 1.
For x ∈ X , the equation ( r ( x ) − θ π )( p ( y = [-+
-]+1 | x ) − γ ) ≥ 0 holds almost everywhere with respect to p ( x ) .
Proof.
By (2) and (12), the following equation also hold, [Equation] [ r [Equation]
x
[Equation]
>
[Equation]
[Equation]
x
[Equation]
[ p
[Equation]
y
[Equation]
+1 | x [Equation] > γ ] p ( x ) d x . Hence, we can derive (cid:90)
[Equation]
This is equivalent to
[Equation]
x = [Equation] From this equation, ( r ( x ) − θ π )( p ( y = [-+
-]+1 | x ) − γ ) ≥ 0 holds almost surely. Using Lemma 1 we prove Theorem 2.
Proof. π [Equation]
[Equation] ≥ γ ] p ( x ) d x is equivalent to [Equation] where this left hand side is equal to
[Equation]
and the right hand side is equal to [Equation] Hence, (12) is equivalent to the following equation, [Equation]
From the deﬁnitions of TP, FP, TN, and FN, the left hand side of the above equation is equal to FP and the right hand side of the above equation is equal to FN .
Hence, FP = FN . We showed FP = FN for a threshold γ , but we can also insist that the same result holds for θ π almost everywhere with respect to p ( x ) by Lemma 1.
Because precision = (cid:16)
[Equation]
means that precision = recall .
C P ROOF OF T HEOREM
Proof.
We ﬁrst consider minimizing R biasPU in the space of all functions from X to [ (cid:15), 1 − (cid:15) ] instead of minimizing it in F . Later, we will see that the minimizer matches f ∗ as stated in Theorem 3 which belongs to F .
The minimization of
[Equation]
over all functions f taking values in [ (cid:15), 1 − (cid:15) ] is reduced to the following point-wise minimization problem
[Equation]
Denoting the solution by z ∗
[Equation]
the Karush-Kuhn-Tucker (KKT) condition of this minimization problem is
[Equation]
where λ and µ are the Lagrange multipliers.
The ﬁrst equation is equivalent to
[Equation]
We investigate the solution by dividing the KKT condition into the following four cases.
If we assume λ = µ = 0 , then the KKT condition is reduced to
[Equation]
In this case, z ∗ ∈ [ (cid:15), 1 − (cid:15) ] is equivalent to x ∈ D 1 ∩ D 2 .
If we assume λ > 0 and µ = 0 , the KKT condition is reduced to
[Equation]
In this case, λ > 0 is equivalent to x (cid:54)∈
D
If we assume λ = 0 and µ > 0 , the KKT condition is reduced to
[Equation]
In this case, λ > 0 is equivalent to x (cid:54)∈ D 1 . 4.
If λ, µ > 0 , there is no feasible solution.
In summary, the solution for the optimization problem arg min z ∈ [
[Equation]
Finally, we deﬁne f ∗ ( x ) :=
arg min z ∈ [ (cid:15), 1 − (cid:15) ]
C ( z, x ) .
It can be conﬁrmed that f ∗ ∈ F because p ( x | y
=
1 , o = +1)
and p ( x ) are measurable and f ∗ takes values in [ (cid:15), 1 − (cid:15) ] . Therefore, the solution of the original optimization problem
[Equation]
is equal to f ∗ almost everywhere.
D N ETWORK S TRUCTURE USED IN S ECTIONS 5.2 AND S ECTIONS 5.
In Section 5.2, we used the MNIST and CIFAR-10 datasets.
The model for the MNIST dataset was a 3 -layer multilayer perceptron (MLP) with ReLU (Nair & Hinton, 2010) (more speciﬁcally, 784-100-1 ).
The model for the CIFAR-10 dataset was an all convolutional net (Springenberg et al., 2015): (32 × 32 ×
[ C (3 × 3 , 96)]
×
2- C (3 × 3 , 96 , 2)-[ C (3 × 3 , 192)]
×
2- C (3 × 3 , 192 , 2)- C (3 × 3 , 192)- C (1 × 1 , 192)- C (1 × 1 , 10)-1000-1000-1 , where the input is a 32 × 32 RGB image, C (3 × 3 , 96) means 96 channels of 3 × 3 convolutions followed by ReLU, [ · ] × 2 means there are two such layers, C (3 × 3 , 96 , 2) means a similar layer but with stride 2, etc.; it is one of the best architectures for CIFAR-10 .
Batch normalization (Ioffe & Szegedy, 2015) was applied before hidden layers.
In Section 5.3, the model for this dataset was a 5 -layer multilayer perceptron (MLP) with ReLU (more speciﬁcally, 78894-300-300-300-300-1 ).
E E XPERIMENTAL R ESULTS OF THE FIRST EXPERIMENT OF S ECTION 5.
In the ﬁrst experiment, we trained a classiﬁer under misspeciﬁed class priors, 0 . 0209(= 0 .
0709 − 0 . 0500) , 0 . 1209(=
0709+0 . 0500) , 0 . 1709(=
0709+0 . 1000) , and 0 . 2709(= 0 .
0709+0 . 2000) .
The results are shown in Figures 4–6.
In Figure 4, the test error in each misspeciﬁed class priors are shown.
In Figure 5, the precision and recall in each misspeciﬁed class prior is shown.
In both the existing method and our method, misspeciﬁed class priors had a bad inﬂuence.
However, our method was less inﬂuenced by the misspeciﬁcation and showed better performance than the existing method.
Besides, the difference between the precision and recall of our algorithm is narrower than that of the existing method as expected.
In order to discuss how misspeciﬁed class prior affects the precision and recall, we show the difference the recall from the precision and show the values in