R ETHINKING C LASS -P RIOR E STIMATION FOR P OSITIVE -U NLABELED L EARNING
A BSTRACT
Given only positive (P) and unlabeled (U) data, PU learning can train a binary classiﬁer without any negative data.
It has two building blocks: PU class-prior estimation (CPE) and PU classiﬁcation; the latter has been well studied while the former has received less attention.
Hitherto, the distributional-assumption-free
CPE methods rely on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution .
If this is violated, those CPE methods will systematically overestimate the class prior; it is even worse that we cannot verify the assumption based on the data.
In this paper, we rethink CPE for PU learning—can we remove the assumption to make CPE always valid?
We show an afﬁrmative answer by proposing Regrouping
CPE (ReCPE) that builds an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution.
ReCPE can work with any CPE method by treating it as the base method.
Theoretically, ReCPE does not affect its base if the assumption already holds for the original probability distribution; otherwise, it reduces the positive bias of its base.
Empirically, ReCPE improves all state-of-the-art CPE methods on various datasets, implying that the assumption has indeed been violated here.
I NTRODUCTION
Positive-unlabeled (PU) learning can date back to 1990s (Denis, 1998; De Comit ´ e et al., 1999;
Letouzey et al., 2000), and there has been a surge of interest in this learning scenario in recent years because of the difﬁculty to annotate large-scale datasets (Ren et al., 2014; du Plessis et al., 2014; 2015; Christoffel et al., 2016; Jain et al., 2016; Ramaswamy et al., 2016; Sakai et al., 2018; Kato et al., 2018; Bekker & Davis, 2018;
Gong et al., 2019; Bai et al., 2021; Xia et al., 2021; Yao et al., 2021).
It is also fallen into different applications, such as knowledge-base completion (Gal ´ arraga et al., 2015; Neelakantan et al., 2015), text classiﬁcation (Lee & Liu, 2003; Li & Liu, 2003), and medical diagnosis (Claesen et al., 2015; Zuluaga et al., 2011).
PU learning can be divided into two different settings based on different data generation processes.
The ﬁrst setting is called censoring PU learning (Elkan & Noto, 2008), which follows a one-sample conﬁguration.
Speciﬁcally, a sample S is randomly drawn from the unlabeled data distribution P u , and a positive sample S p is then distilled from it, i.e., randomly selecting some positive instances contained in the unlabeled data to be the positive sample.
The second setting is called case-control
PU learning (Kiryo et al., 2017).
In this setting, a positive sample S p = { x i } ki =1 is randomly drawn from the positive class-conditional distribution P p
=
P
( X | Y = 1) , and an unlabeled sample
S u = { x i } ni = k +1 is randomly drawn from the unlabeled data distribution P u .
Because case-control
PU learning is more general than censoring PU learning (Niu et al., 2016), therefore, we will focus on the setting of case-control PU learning.
Under the setting of case-control PU learning, a lot of classiﬁcation methods have been proposed (Ren et al., 2014; du Plessis et al., 2014; 2015; Christoffel et al., 2016; Sakai et al., 2018;
Kato et al., 2018; Bekker & [Figure]
However, the class-prior estimation (CPE) (Elkan & Noto, 2008; Jain et al., 2016; Ramaswamy et al., 2016; Christoffel et al., 2016; Kato et al., 2018) has received less attention.
Formally, CPE is deﬁned as a problem of estimating π = P ( y = 1) ∈ (0 , 1) given a sample from the marginal distribution P u and a sample from positive class-conditional distribution P p .
The marginal distribution P u is mixed with both positive and negative class-conditional distributions, i.e., P u = πP p
+ (1 − π ) P n .
CPE acts as a crucial building block for state-of-the-art PU classiﬁcation methods, and it is essential to build statistically-consistent PU classiﬁers (du Plessis et al., 2014; Scott, 2015;
Jain et al., 2016; Kiryo et al., 2017).
The formulation of these classiﬁcation methods involves the class-prior π , but π is usually unknown in practice.
If π is poorly estimated, the classiﬁcation accuracy of the state-of-the-art PU classiﬁcation methods (du Plessis et al., 2014; 2015; Kiryo et al., 2017) could be degraded.
The mixture proportion estimation (MPE) is closely related to CPE (Blanchard et al., 2010; Scott, 2015).
In the setting of MPE, there is a mixture distribution
[Equation]
where H and G are called component distributions.
Given the samples randomly drawn from
F and H , respectively, MPE aims to estimate the maximum proportion κ ∗ ∈ (0 , 1) of H in F .
Thereby, if the maximum proportion κ ∗ is identical to the class-prior π , the MPE methods can be employed to obtain π by letting P u and P p be the mixture distribution F and the component distribution H , respectively; otherwise, the MPE methods cannot be employed.
To the best of our knowledge, most of state-of-the-art CPE methods (Blanchard et al., 2010; Liu & Tao, 2015; Scott, 2015; Ramaswamy et al., 2016; Jain et al., 2016) are based on MPE, which do not rely on assumptions that the data are drawn from a given parametric family of probability distributions (i.e., they are distributional-assumption-free methods).
To let these distributional-assumption-free methods can be used to identify class-prior π , κ ∗ must be identical to the class-prior π .
The irreducibility assumption (Blanchard et al., 2010) has been proposed to make them identical, which is employed by all these CPE methods implicitly or explicitly, to the best of our knowledge.
It assumes that the support of the positive class-conditional distribution
P p is not contained in the support of the negative class-conditional distribution P n .
However, it is strong and hard to be veriﬁed in PU learning, since P n is a latent distribution, such that we do not have any prior knowledge about it.
Additionally, since the applications of PU learning are diverse (Hsieh et al., 2019; Bekker & Davis, 2020), it is hard to guarantee that the support of P p is not in the support of P n .
If the irreducibility assumption cannot be satisﬁed, the existing distributional-assumption-free CPE methods will suffer from an overestimation of π .
For example, in Figure 1(a), we show both the unlabeled data distribution P u and the component distribution P p .
In Figure 1(b), we assume the latent negative class-conditional distribution P n is ﬁxed as shown in the green color, and the positive class-prior π = 0 .
In Figure 1(c), we show the existing distributional-assumption-free CPE methods will output the biased class-prior 0 .
It is different from the ground truth 0 .
5 , since the support of P p is contained in the support of P n .
When the irreducibility assumption is not held, how to improve the estimations of distributional-assumption-free PU learning methods is challenging but useful.
Because the irreducibility assumption is impossible to check without making any assumption on
P n .
Thereby, in this paper, we rethink those CPE methods and propose a novel method called
Regrouping CPE (ReCPE) which improves the estimations of the current PU learning methods without irreducibility assumption.
The main idea of our method is that, instead of estimating the maximum proportion of P p in P u , we build a new CPE problem by creating a new auxiliary distribution P p (cid:48) always guaranteeing the irreducibility assumption.
Then we use the existing CPE method to obtain the maximum proportion of P p (cid:48) in P u , which is denoted by π (cid:48) .
We show that, with both theoretical analyses and experimental validations, when the irreducibility assumption holds, our ReCPE method does not affect the prediction of the existing estimators; when the irreducibility assumption does not hold, our method will help the current estimators have less estimation bias, which could improve the performances of PU classiﬁcation tasks.
For example, in Figure 1(d), we create a new class-conditional (auxiliary) distribution P p (cid:48) .
By solving it, π (cid:48) = 0 .
The estimation bias of the existing estimators will reduce to π (cid:48)
− π = 0 .
01 instead of κ ∗ − π = 0 .
The rest of the paper is organized as follows.
In Section 2, we review the irreducibility assumption and its variants.
We discuss the difﬁculty of checking the assumptions.
In Section 3, we provide the estimation biases of the existing consistent distributional-assumption-free CPE methods.
Then we propose our method ReCPE, followed by theoretically analysis of its estimation bias and the implementation details.
All the proofs are listed in Appendix A.
The experimental validations are given in Section 4.
Section 5 concludes the paper.
I RREDUCIBILITY OF CPE
In this section, we brieﬂy review the assumptions used for existing distributional-assumption-free CPE estimators.
Then we provide the estimation bias introduced by consistent distributional-assumptionfree CPE methods when the assumptions do not hold.
The irreducibility assumption.
Let P p and P u be probability measures (distributions) on a measurable space ( X , S ) , where X is the sample space, and S is the σ -algebra.
Let κ
∗
be the maximum proportion of P p in P u .
To let κ ∗ be identical to π , the irreducibility assumption was proposed by
Blanchard et al.
Deﬁnition 1 (Irreducibility) .
P n and P p are said to satisfy the irreducibility assumption if P n is not a mixture containing P p .
That is, there does not exist a decomposition P n = (1 − β ) Q + βP p , where
Q is a probability distribution on the measurable space ( X , S ) , and 0 < β ≤ 1 .
Equivalently, the assumption assumes the support of P p is hardly contained in the support of P n .
It means that with the selection of different sets S , the probability P n ( S ) can be arbitrarily close to 0 , and P p ( S ) > 0 .
Suppose we can access the distributions P u , P p and the set C containing all possible latent distributions, then the class-prior π can be found as follows:
[Equation]
To the best of our knowledge, all existing distributional-assumption-free CPE methods (Blanchard et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015; Ramaswamy et al., 2016; Ivanov, 2019) are variants of estimating the maximum proportion κ ∗ of P p in P u .
Many of them are statistically consistent estimators (Blanchard et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015).
The variants of the irreducibility.
Based on the irreducibility assumption, estimators can be designed with theoretical guarantees that they will converge to the class-prior π
(Blanchard et al., 2010).
However, the convergence rate can be arbitrarily slow (Scott, 2015).
The reason is that the irreducibility assumption implies the following fact (Blanchard et al., 2010; Scott et al., 2013)
[Equation]
i.e., the maximum proportion of P p in P n approaches to 0 .
To obtain the class-prior π , it requires ﬁnding a sequence of the sets S converging to the inﬁmum, which empirically can be hard to ﬁnd.
Therefore, the convergence rate of the designed estimators based on Eq.
(3) will be arbitrarily slow.
To ensure a ﬁxed rate of convergence, the anchor set assumption, a stronger variant of the irreducibility assumption, has been proposed (Scott, 2015; Liu & Tao, 2015; Xia et al., 2019; 2020; Yao et al., 2020).
It assumes that
[Equation]
i.e., there exists a set can achieve the minimum 0 , which is called an anchor set.
Another stronger variant is the separability assumption (Ramaswamy et al., 2016) which extends the anchor set assumption to a function space.
It is proposed to bound the convergence rate of the method based on kernel-mean-matching (KMM) technique (Gretton et al., 2012).
R EGROUPING FOR CPE (R E CPE)
In this section, we propose a general method named regrouping for CPE (ReCPE).
We discuss how to theoretically and empirically mitigate the overestimation problem of the class-prior π .
3. M OTIVATION
In general, it is impossible to verify the irreducibility assumption for CPE.
To check the assumption, we need to make P n itself to be observable and verify that whether the distribution P n is a mixture containing the distribution P p , which obviously contradicts the setting of PU learning.
However, in practice, the irreducibility assumption may not hold for many real-world problems, because the negative class is diverse (Hsieh et al., 2019; Bekker & Davis, 2020) in PU learning.
If the assumption does not hold, P n is said to be reducible to P p , and distributional-assumption-free CPE methods will introduce an estimation bias .
Proposition 1.
Let β ∗ =
inf S ∈ S ,P p ( S ) > 0 P n ( S ) P p ( S ) be the maximum proportion of P p in P n , given
P u
= (1 − π ) P n + πP p , for 0 < π ≤ 1 , we have
[Equation]
According to Proposition 1, if the irreducibility assumption does not hold, then there exists β > 0 .
In this case, maximum proportion κ ∗ can still be obtained, but it is different from π but equal to π + (1 − π ) β ∗ .
In this case, if we directly employ existing distributional-assumption-free CPE methods, they could introduce an arbitrary estimation bias (1 − π ) β ∗ which depends on P n .
To reduce the estimation bias, we propose ReCPE.
The process of regrouping is to change the original class-conditional distributions P n and P p into new class-conditional distributions P n (cid:48) and P p (cid:48) by transporting the probability mass of the set
A from the negative class to the positive class.
After regrouping, new class-conditional distributions are guaranteed to satisfy the irreducibility assumption, and therefore, the new positive class-prior π (cid:48) can be identiﬁed by current CPE methods.
To get the intuition, we provide a concrete example as follows.
Suppose that P p is the uniform on [ 12 , 1] , P n is uniform on [0 , 1] , and π = 12 .
Then we have P u such that it is uniform on [0 , 12 ) and [ 12 , 1] , respectively.
Speciﬁcally, the probabilities are
[Equation]
In this case, by Eq. 5, the maximum proportion of P p in P u is π ∗ = 34 .
Let ρ > 0 be a small constant and let
A =
(1 − ρ, 1] .
In this case, the mass of A in P u from P n is πP n ( A )
=
ρ 2 .
After transporting the mass ρ 2 from P n to P p , we have a new positive class-prior π (cid:48) and a new class-conditional P p (cid:48) which is uniform on [ 1 2 , 1 − ρ ) and [1 − ρ, 1] , respectively.
Speciﬁcally,
[Equation]
The left equation above shows that the new class-prior π (cid:48) is dependent on ρ or the size of A .
By controlling set A or ρ to be small, π (cid:48) can be as close to π as possible.
This is the intuition of how regrouping works.
3. P RACTICAL I MPLEMENTATION
In practice, we have to implement the aforementioned idea of regrouping based on positive sample
S p and unlabeled sample S u .
Since the negative sample is unavailable, we cannot “cut and paste” any example from negative class to positive sample S p ; instead, we can “copy and past” some unlabeled examples to S p .
When doing so, we should select a small set of samples ˆ A ∗ which look the most similar to the positive class and dissimilar to the negative class, which could encourage the difference between the original P n , P p , and ρ and π , P p (cid:48) , and π (cid:48) to be small.
This is why ˆ A ∗ =
(1 − ρ, 1] was selected in the above intuitive example, i.e., ˆ A ∗ belongs geometrically and visually to the positive class with the highest conﬁdence among all subsets of [0 , 1] of size ρ .
A hyper-parameter p ∈ (0 , 1) is introduced to control the size of set ˆ A ∗ , theoretically, we prefer the set ˆ A ∗ to have a small size.
Empirically, p cannot be so small: the existing estimators are insensitive to tiny modiﬁcations (they are designed to be robust in such a way, in order to be good estimators).
For example, the difference between the estimated class-priors by employing samples S u and S p and the one by employing samples S u and S p (cid:48) can be hardly observed if S p and S p (cid:48) only differ from in one or two points.
Speciﬁcally, p = 10% is selected for the experiments on all datasets, which leads to a signiﬁcant improvement of the estimation accuracy.
The details on the selection of the hyper-parameter value will be explained in Section 4.1.
The algorithm is summarized in Algorithm 1.
There are two fundamental concerns for copying ˆ A ∗ to S p . 1).
When we have irreducibility, might regrouping make ˆ π (cid:48) be a worse approximation? 2).
When we lack irreducibility, must regrouping make ˆ π (cid:48) be a better approximation?
While these concerns will be formally clariﬁed later, we give here intuitive implications of regrouping.
If we have irreducibility, the P n ( ˆ A ∗ ) should be rather small (if not zero), and ˆ A ∗ should be drawn from the positive component P p of the mixture P u .
In this case, regrouping will generally have small inﬂuence to P p .
Hence, it will not make ˆ π (cid:48) worse.
If we lack irreducibility, ˆ A ∗ may be drawn from either P p or P n .
By regrouping, ˆ A ∗ becomes present in S p (cid:48) , which encourages the probability of the set ˆ A ∗ in P p (cid:48) to be large.
This will modify P p as we expected towards irreducibility.
As a consequence, regrouping will make ˆ π (cid:48) better.
3. T HEORETICAL J USTIFICATION
In the regrouping approach described above, the auxiliary class-conditional distribution P p (cid:48) and
P n (cid:48) are created by regrouping a small set
A from P p and P n .
Here, we analyze the properties of regrouping and theoretically justify it.
A formal deﬁnition of regrouping In order to analyze the properties, we need to formally deﬁne how to split, transport, and regroup a set A (or the mass of A ).
Deﬁnition 2.
Let M be a probability measure on a measurable space ( X , S ) .
Given a set A ∈ S , we deﬁne a measure M A on the σ -algebra S as follows: [Equation]
[Equation]
It is easy to see that given two measures M A and M A c obtained according to Deﬁnition 2, where
A c = X \ A , then M A and M A c have the following property.
Lemma 1.
Let M be a probability measure over a measurable space ( X , S ) .
For any set A ∈ S , we have M A + M A c = M .
Now, we introduce the theory of regrouping.
Fixing a set A ∈ S , we split P n as P
A c n and P A n , transport P A n to P p to regroup them together, i.e.,
[Equation]
Finally, we can rewrite the unlabeled data distribution P u as a mixture of two new class-conditional distributions P n (cid:48) and P p (cid:48) deﬁned in Theorem 1 by normalization.
Theorem 1.
Let P u
= (1 − π ) P n + πP p .
Let A ⊂ support( P u ) .
By regrouping P A n to P p , P u can be written as a mixture, i.e., P u
= (1 − π
[Equation]
) P n (cid:48)
+ π (cid:48)
P p (cid:48) , where
[Equation]
and P n (cid:48) and P p (cid:48) satisfy the anchor set assumption.
When class-conditional distributions P n and P p do not satisfy the irreducibility assumption, π cannot be obtained by using CPE methods based on MPE, which will lead to an estimation bias discussed before.
However, Theorem 1 shows that the new proportion π (cid:48) is always identiﬁable as P n (cid:48) and
P p (cid:48) always satisfy the anchor set assumption.
Thus, after regrouping, π (cid:48) is identiﬁable and can be estimated by the existing CPE methods.
Bias reduction According to Theorem 1, to make π (cid:48) closer to π , we expect to ﬁnd the set A looks most dissimilar to the negative class, i.e., P n ( A ) is small.
Theorem 2.
Let P p (cid:48) and P n (cid:48) be obtained by regrouping a set A ∗ :=
arg min
A ∈ S P n ( A ) P p ( A ) 1 from P p and P n . 1).
If P n and P p satisfy the irreducibility assumption, then π (cid:48)
=
π ; 2).
if P n and P p dissatisfy the irreducibility assumption, then π < π (cid:48) <
π + (1 − π ) β ∗ = κ ∗ .
Theorem 2 shows how to properly select a set used for regrouping to make π (cid:48) a good approximation of π .
Speciﬁcally, once A ∗ is selected for regrouping, if P n and P p satisfy the irreducibility assumption, the new estimation π (cid:48) will be identical to π ; if P n and P p dissatisfy the irreducibility assumption, π (cid:48) obtained by employing the distributions P u and P p (cid:48) will contain a smaller estimation bias compared to κ ∗ obtained by employing the distributions P u and P p .
Convergence analysis For completeness, we illustrate the convergence property of ReCPE, which is presented by employing the estimator proposed by Blanchard et al.
Let S u , S p and S p (cid:48) be the samples i.i.d.
drawn from P u , P p and P p (cid:48) , respectively.
Let A be the set used for regrouping.
Let h : X → R , h ∈ H , be a function that predicts 1 for all elements in the set A and 0 otherwise, where H denotes a hypothesis space .
Let | S | denote the cardinality of a set S. Let 1 { h ( x )=1 } be an indicator function which returns 1 if h ( x ) predicts 1 and 0 otherwise.
Then P u ( A ) can be expressed as (cid:82) x ∈X p p ( x ) 1 { h ( x )=1 } d x , where p p is the density function of the distribution P u .
Let ˆ P u
( A ) be the empirical version of P u
( A ) , i.e., ˆ P u ( A ) = 1 | S u | (cid:80) x ∈X 1 { h ( x )=1 } .
Similarly, let ˆ P p (cid:48)
( A ) be the empirical version of P p (cid:48)
( A ) .
Let the error (cid:15) δ, H ( S u ) denote the difference between P u ( A ) and ˆ P u ( A ) obtained by exploiting the empirical Rademacher complexity (Mohri et al., 2018).
Similarly, let (cid:15) δ, H ( S p (cid:48) )
denote the difference between P p (cid:48)
( A ) and ˆ P p (cid:48)
( A ) .
We have the following theorem.
Theorem 3.
Let P u
= (1 − π ) P n + πP p .
By selecting a set A and regrouping P A n to P p .
Then, with probability 1 − 2 δ , the estimated class-prior ˆ π (cid:48) based on solving inf S ∈ S , ˆ P p (cid:48) ( S ) >
0 ˆ P u ( S )
ˆ P p (cid:48)
( S ) satisﬁes
| ˆ π (cid:48) − π | ≤ (cid:15) δ, H ( S p (cid:48) )
ˆ P p (cid:48)
( A ) + (cid:15) δ, H ( S p (cid:48)
[Equation]
[Equation] where (cid:15) δ, H ( S ) (cid:44) {+
+} ˆ R S
( H ) + 3 (cid:113) log 4 δ 2 | S | , and ˆ R S ( H ) is the empirical Rademacher complexity of H .
To make (cid:15) δ, H ( S ) converge to 0 with the increasing of the sample size of S , a universal approximation assumption has been proposed by Scott (2015) to ensure that the hypothesis space is large enough to represent a wide variety of interesting functions.
Under the assumption, Scott (2015) proved that, with increasing of the size of samples S u and S p (cid:48) , the error between P u ( A ) and ˆ P u ( A ) will converge to 0 at a rate O (cid:16)(cid:113) log | S u | | S u | (cid:17) , similarly to P p (cid:48)
( A ) and ˆ P p (cid:48)
( A ) .
Since the empirical Rademacher complexity ˆ R X
( H ) of a hypothesis space H can be upper-bounded by its VC-dimension (Mohri et al., 2018), the both errors based on the empirical Rademacher complexity will also converge to zero with increasing of the sample size.
Consequently, the estimation ˆ π (cid:48)
=
ˆ P u ( A ) ˆ P p (cid:48)
( A ) will converge to π (cid:48)
=
P u ( A ) P p (cid:48) ( A ) = π + (1 − π ) P n ( A ) at a rate O (cid:18)(cid:114) log(min( | S
[Equation]
|
[Equation]
| S
[Equation]
| ))
min( | S u | , | S p (cid:48) | ) (cid:19) .
Computationally efﬁcient identiﬁcation of A ∗ The following theorem presents how to identify A ∗ with P u and P p .
Let us deﬁne another auxiliary distribution q ( X, C ) , where C ∈ { 0 , 1 } is the positivevs-unlabeled label i.e., a class label distinguishing between the positive component and the whole mixture.
Speciﬁcally, priors are q ( C = 1) :
= π 1 − π and q ( C = 0) :
= 1 1 − π ; conditional densities are q ( X | C = 1) :
= P p and q ( X | C = 0) := P u ; class-posterior probabilities are q ( C = 0 | X ) and q ( C = 1 | X ) .
We have the following theorem.
Theorem 4.
Let p u and p p be density functions of P u and P p , respectively.
Let q = P
( C = 0)
p u + P ( C = 1) p p .
Let 1 A : X → { 0 , 1 } be the identity function which outputs 1 if x ∈ X is in the set A , and 0 otherwise.
Then the set A ∗ =
[Equation]
For the above optimization, its objective function has two expectations over q , which can have the “exact” empirical solution obtained by replacing expectations with empirical averages: A ∗ =
[Equation]
p (cid:48) with a surrogate As we do not have examples drawn from P n , it is hard to create P p (cid:48) , let alone sample from it.
We approximate P p (cid:48) by using P ˜p (cid:48)
=
P A u + P p P u ( A )+1 .
The following proposition shows that when P u ( A ) is small, P ˜p (cid:48) is almost identical to P p (cid:48) .
Proposition 2.
Let P ˜p (cid:48)
=
P A u + P p P
u
( A )
+1 and P u ( A ) < (cid:15) .
∀ (cid:15) > 0 and S ∈ S , | P p (cid:48) ( S ) − P ˜p (cid:48)
( S )
| ≤ O ( (cid:15) ) .
Since the gap has the same order as P u ( A ) uniformly over S , it is guaranteed that whenever P u ( A ) is small, the gap is also small.
Practically, we can control the parameter (cid:15) in the above proposition to be small.
Speciﬁcally, using a small value of the hyper-parameter p in Algorithm 1 will lead to the set A in Theorem 1 to be small, as well as P u
( A ) .
As a consequence, the practical implementation of regrouping is a good approximation of the theory of regrouping as we expected.
By now, we have analyzed all of the properties of regrouping and theoretically justiﬁed all of the points in its design.
E XPERIMENTS
We run experiments on 2 synthetic datasets and 9 real word datasets 2 .
The objectives of employing synthetic datasets are to validate whether the proposed regrouping CPE method reduces the estimation error of the consistent distributional-assumption-free CPE method on the dataset satisfying the irreducibility assumption and does not inﬂuence the prediction of the CPE method on the dataset dissatisfying the irreducibility assumption.
The hyper-parameter p is also selected from the synthetic datasets.
The real-world datasets are used to illustrate the effectiveness of our methods.
Although we have introduced a hyper-parameter p and used approximations in the implementation, empirical results on all synthetic and real-world datasets consistently show the superiority of ReCPE.
To have a rigorous performance evaluation, for each dataset, 6 × 3 × 10 experiments are conducted via random sampling.
Speciﬁcally, we select { 0 . 25 , 0 . 5 , 0 .
75 } fraction of positive examples to be the sample of the positive distribution P p .
We let the rest of the examples be the sample of the unlabeled distribution P u .
In such a way, 3 pairs of empirical positive and unlabeled distributions are generated.
Then, we create other 3 pairs of distributions by ﬂipping the labels of all instances in the original datasets.
For each pair of distributions, we randomly draw positive and unlabeled samples with sizes of 800 , 1600 , and 3200 , respectively, which are used as input data.
Note that, the positive and unlabeled samples have the same size as did in Ramaswamy et al.
For each sample size,repeated experiments are carried out with random sampling.
For all experiments, we employ a neural network 3 with 2 hidden layers.
Each hidden layer contains 50 hidden units.
The batch normalization (Ioffe & Szegedy, 2015) is also employed.
The stochastic gradient descent optimizer is used with the batch size 50 .
The network is trained for 350 epochs with a learning rate 0 . 01 and momentum 0 .
The weight decay is set to 1 e − 5 .
The model with the best validation accuracy is used to estimate the positive class-posterior probability P ( Y = 1 | X = x ) .
We sample the validation set with 20% of the training data size.
4. E XPERIMENTS ON S YNTHETIC D ATASETS
[Figure]
- grouping KM2 (ReKM2) and (b) average differences of the absolute error between KM2 and Regrouping
KM2 (ReKM2).
We create two datasets with one satisfying the irreducibility assumption while the other not.
The dataset satisfying the irreducibility assumption is created by sampling fromdifferent 10-dimensional Gaussian distributions as the component distributions.
One of the distributions has zero means and a unit covariance matrix.
Another one has unit means and unit covariance matrix.
The dataset dissatisfying the irreducibility assumption is also created by drawing examples from 2 different 10-dimensional Gaussian distributions.
One of the distributions has zero means and unit covariance matrix.
Another one has unit means and covariance matrix.
Then we remove all the data points with P ( Y = 1 | X ) ≥ 0 .
98 or P ( Y = 1 | X ) ≤ 0 .
For simplicity, in Figure 2, we name two datasets irreducible data and reducible data, respectively.
To validate the correctness of our method and to select a suitable value of the hyper-parameter p , we carry out two experiments.
The consistent CPE method KM2 is used as the baseline, which is compared to our method ReKM2, i.e., regrouping version of the KM2.
Firstly, we compare the magnitude differences between ˆ π and ˆ π (cid:48)
(i.e., ˆ π − ˆ π (cid:48) ) with the different fractions of points to be copied from the mixture sample to the component sample, which is illustrated in Figure 2(a).
Then we compare differences of the absolute error (i.e., | ˆ π − π | − | ˆ π (cid:48) − π | ) between the baseline and our method with the increasing of the copy fractions.
Note that each point in Figure 2 is obtained by averaging over 6 × 3 × 10 experiments.
Figure 2(a) validates the correctness of our Theorem 2 and
Eq.
Theorem 2 states that, by properly selecting the set A , on the dataset dissatisfying the irreducibility assumption (reducible data), π (cid:48) should be smaller than the maximum proportion κ ∗ ; on the dataset satisfying the irreducibility assumption (irreducible data), π (cid:48) should be close to π .
Figure 2(a) perfectly matches this statement.
It shows that, on the reducible data, the values of ˆ π (cid:48) are continuously smaller than ˆ π with the copy fraction ≤ 22 .
5% ; on the irreducible data, ˆ π (cid:48)
and ˆ π have the similar values until the copy fraction ≥ 17 .
According to Eq.
(7), the positive bias of our estimator should become larger with the increase of P n ( A ) .
This fact is reﬂected by the differences of ˆ π − ˆ π (cid:48) become smaller on both datasets when the copy fraction > 15% .
Figure 2(b) illustrates the average differences of absolute error between the baseline and the proposed method.
On the reducible data, our method continuously outperforms the baseline with the copy
[Table]
fraction ≤ 22 . 5% . However, the differences of average absolute error start to decrease with the copy fraction > 15% . On the irreducible data, the differences of average absolute error are close to zero until the copy fraction > 15% .
By observing Figure 2, we found the prediction of the KM2 estimator will not change much if the copy fraction p is too small.
For example, the difference between the estimated mixture proportion by employing samples S u and X P and the ones by employing samples S u and X (cid:48)
P can be hardly observed if X P and X (cid:48)
P only differ from in one or two points.
For simplicity and consistency, we select hyper-parameter p to be 10% for all the following experiments
. 4. E XPERIMENTS ON R EAL - WORLD D ATASETS
We illustrate the absolute estimation errors of different estimators on the real-world datasets.
Totally, 7 baseline methods are used in the experiments, which are AlphaMax (AM) (Jain et al., 2016),
DEDPUL (DPL) (Ivanov, 2019), Elkan-Noto (EN) (Elkan & Noto, 2008), KM1, KM2 (Ramaswamy et al., 2016), ROC (Scott, 2015), and Rankpruning (RPG) (Northcutt et al., 2017).
By using our method, the regrouped version of them are implemented, which are called ReAM, ReDPL, ReEN,
ReKM1, ReKM2, ReROC, and ReRPG.
In Table 1, we compare the absolute estimation errors of each baseline with those of its regrouped version on different datasets with different sample lengths.
Each number in Table 1 is the average over 6 × 10 experiments.
Table 1 reﬂects the effectiveness of our regrouping CPE method.
Overall, by using our method, the estimation accuracy is increased for most of the popular CPE methods among most of the datasets with different sample lengths.
By observing the last row, the regrouped version of the estimators has much smaller average estimation errors except DPL, KM2, and RPG.
On the real-world datasets,
Regrouping AlphaMax (ReAM) has the smallest average estimation error among all methods.
C ONCLUSION
In this paper, we investigate how to reduce the estimation bias of the distributional-assumption-free
CPE method without irreducibility assumption for PU learning.
We have proposed regrouping CPE which can be employed on top of most existing CPE methods.
We have also theoretically analyzed the estimation bias of ReCPE.
Empirically, it improves all popular CPE methods on various datasets.
One future work will focus on how to generate a sample from P p (cid:48) instead of using an approximation.
R EPRODUCIBILITY S TATEMENT
For theoretical results, we have clearly explained any assumptions.
A complete proof of the claims can be founded in the appendix.
We have also included an anonymous source code in our supplementary material.
For any datasets used in the experiments, a complete description of the data processing steps is provided In Section
A CKNOWLEDGMENTS
TL was partially supported by Australian Research Council Projects DP180103424, DE-190101473,
IC-190100031, and DP-220102121.
BH was supported by the RGC Early Career Scheme No.
22200720 and NSFC Young Scientists Fund No. 62006202.
MG is supported by ARC DE210101624.
GN and MS were supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, Japan.
MS was also supported by the Institute for AI and Beyond, UTokyo.