W HO I S Y OUR R IGHT M IX U P P ARTNER IN P OSITIVE AND U NLABELED L EARNING
A BSTRACT
Positive and Unlabeled (PU) learning targets inducing a binary classiﬁer from weak training datasets of positive and unlabeled instances, which arise in many real-world applications.
In this paper, we propose a novel PU learning method namely Positive and unlabeled learning with
Partially
Positive
Mixup (
P 3 M ix
which simultaneously beneﬁts from data augmentation and supervision correc- tion with a heuristic mixup technique
To be speciﬁc we take inspiration from the decision boundary deviation phenomenon observed in our preliminary experiments where the learned PU boundary tends to deviate from the fully supervised boundary towards the positive side
For the unlabeled instances with ambigu
- ous predictive results, we select their mixup partners from the positive instances around the learned PU boundary
, so as to transform them into augmented instances near to the boundary yet with more precise supervision.
Accordingly, those aug
- mented instances may push the learned PU boundary towards the fully supervised boundary, thereby improving the classiﬁcation performance.
Comprehensive experimental results demonstrate the effectiveness of the heuristic mixup technique in
PU learning and show that P 3 M ix can consistently outperform the state
-of-the- art PU learning methods.
I NTRODUCTION
P ositive and U nlabeled ( PU ) learning refers to a speciﬁc binary classiﬁcation problem, where only a small number of positive training instances are manually annotated but all other instances are unlabeled (Liu et al., 2002).
Such kind of datasets naturally arise in many signiﬁcant real-world scenarios such as product recommendation (Hsieh et al., 2015), deceptive reviews detection (Ren et al., 2014), and medical diagnosis (Yang et al., 2012).
For speciﬁc example, many diseases, e.g.,
Alzheimer’s disease, Amyotrophic Lateral Sclerosis, and Parkinson’s disease, are very infrequent and with long latency, hence only few diagnosed patients are known but a much larger population of undiagnosed individuals may be either diseased or healthy.
Treating the diagnosed ones as positive instances and the undiagnosed ones as unlabeled instances results in such PU datasets of medical diagnosis.
To meet those practical demands, PU learning has drawn increasing interest from the machine learning community (Bekker & Davis, 2020).
Formally, let x ∈ R d and y ∈ { 0 , 1 } be the feature representation and category label, respectively, where the positive instance is indicated by y = 1 and the negative one by y = 0 .
In the context of PU learning, the training dataset is composed of the sets of positive instances P = { ( x i , y i = 1) } n p i = and unlabeled instances U
= { x i } n p + n u i = n p +1 , where U contains both positive and negative instances.
The target is to learn a binary classiﬁer based on such weak training dataset P ∪ U .
During the past decades, many PU learning methods have been proposed, where, naturally, the essential idea is to estimate the negative instances from the set of unlabeled instances U .
Generally, most of existing PU learning methods can be divided into two categories, termed as sample-selection methods and cost-sensitive methods.
The sample-selection methods, as the name suggests, mainly select reliable negative instances from U using various heuristic strategies,
e.g., Na¨ıve Bayes (Liu et al., 2002), k NN (Zhang & Zuo, 2009), k -means (Chaudhari & Shevade, 2012), and reinforcement learning (Luo et al., 2021); and then apply supervised methods over positive and those reliable negative instances.
In contrast, the cost-sensitive methods treat all unlabeled instances as corrupted negative ones, and correct the estimation bias of the objective by employing well-designed misclassiﬁcation risks such as unbiased risk estimator (du Plessis et al., 2014; 2015; Kiryo et al., 2017) and maximum margin loss (Shi et al., 2018;
Gong et al., 2019b; Zhang et al., 2019;
Gong et al., 2019a).
Orthogonal to the aforementioned techniques, we note that some PU learning methods such as (Chen et al., 2020a; Wei et al., 2020) have made preliminary attempts to integrate with the art of mixup, i.e., an economic-yet-effective data augmentation method (Zhang et al., 2018).
Formally, mixup generates an augmented instance ( (cid:98) x , (cid:98) y ) with the convex combination of any pair of instances { ( x i , y i ) , ( x j , y j ) } drawn from the training dataset:
[Equation]
Previous studies have indicated that mixup is approximately equivalent to applying adversarial training (Zhang et al., 2021), enabling to improve robustness with even scarce and noisy supervision (Thulasidasan et al., 2019; Carratino et al., 2020; Zhang et al., 2021).
Accordingly, it has been successfully used to solve various learning problems with weak supervision, e.g., semi-supervised learning (Berthelot et al., 2019), noisy label learning (Li et al., 2020b), and partial label learning (Yan & Guo, 2020).
Our story and contribution.
Inspired by the recent success of mixup in learning with weak supervision, our original goal is to thoroughly investigate the impact of mixup in PU learning.
To this end, we begin with a naive disambiguation-free objective of PU learning, where all unlabeled instances are treated as pseudo-negative instances , denoted by (cid:101) U
= { ( x i , y i = 0) } n p + n u i = n p +1 , and the binary classiﬁer is trained based on P ∪ (cid:101) U .
In preliminary experiments, we found an interesting phenomenon, where the number of training instances predicted as positive by the disambiguationfree classiﬁer tends to be smaller than usual as illustrated in Fig.1.
This phenomenon implies that the disambiguation-free boundary tends to deviate from the fully supervised boundary towards the positive side, expressed by a toy example shown in Fig.2(a).
We consider that the decision boundary deviation is mainly caused by the marginal pseudo-negative instances , which lie between the two boundaries.
Such instances are more likely to be positive but actually annotated by negative.
Motivated by this observation, we extend mixup to a speciﬁc heuristic version for PU learning, enabling to achieve data augmentation and supervision correction simultaneously.
Its basic idea is to transform the marginal pseudo-negative instances into augmented instances which are partially positive and yet also lie between the two boundaries, so as to push the learned boundary towards the fully supervised one.
This can be achieved by selecting the mixup partners for marginal pseudonegative instances from the positive instances that are around the learned boundary, as expressed in
Fig.2(b).
With this insight, we propose a novel PU method, namely P ositive and unlabeled learning with P artially P ositive Mix up ( P 3 M ix ).
Generally, P 3 M ix is easy-to-implement, where, speciﬁcally, we can deﬁne the marginal pseudo-negative instances using the predictive results and the positive instances around the boundary using the entropy values of predictive results.
To evaluate the effectiveness of P 3 M ix, we conduct a number of experiments on benchmark datasets.
Experimental results demonstrate that P 3 M ix can consistently outperform the state-of-the-art PU methods.
T HE PROPOSED P 3 MIX METHOD
In this section, we introduce the proposed P 3 M ix method for PU learning.
We ﬁrst revisit and clarify some important notations: the set of positive instances P = { ( x i , y i = 1) } n p i =1 and the set of unlabeled instances U = { x i } n p + n u i = n p
+1 .
By treating all unlabeled instances as negative, we translate
U into the set of pseudo-negative instances (cid:101) U
= { ( x i , y i = 0) } n p + n u i = n p +1 .
Given batches X p ⊂ P and
X u ⊂ (cid:101) U , the disambiguation-free objective of PU learning can be formulated as follows:
[Equation]
where f ( · ; Θ ) is a trainable neural network, i.e., the binary classiﬁer, parameterized by Θ ; (cid:96) ( · , · ) is the loss function; and β is the coefﬁcient parameter.
To achieve data augmentation and supervision correction simultaneously, P 3 M ix transforms X p and
X u into the batches of augmented instances (cid:98) X p and (cid:98)
X u using the proposed heuristic mixup technique.
Accordingly, the objective of P 3 M ix is then expressed as follows:
[Equation]
where α ∈ (0 , ∞ ) is a hyperparameter of mixup.
Next, we describe the details of heuristic mixup.
; Θ j j
X j =
2. T RAINING WITH H EURISTIC M IXUP
Basically, for each instance ( x i , y i ) ∈
X p ∪ X u we select a mixup partner ( x j , y j ) to generate an augmented instance ( (cid:98) x i , (cid:98) y i ) using the modiﬁed mixup operator 2 (Berthelot et al., 2019):
[Equation]
accordingly forming the augmented instance sets (cid:98) X p and (cid:98) X u .
Our heuristic mixup refers to a guidance of mixup partner selection to reﬁne the imprecise supervision within X u .
We take inspiration from the phenomenon, where the boundary learned by
Eq.(1) tends to deviate from the fully supervised boundary towards the positive side as illustrated in
Fig.2(a).
The marginal pseudo-negative instances X mpn ⊂
X u lie between the two boundaries, and they are more likely to be positive but actually annotated by negative.
To resolve this problem, for each of them we uniformly select a mixup partner from the candidate mixup pool X cnd ⊂ P of positive instances that are around the current learned boundary, so as to generate an augmented instance which is partially positive and yet also lies between the two boundaries as expressed in Fig.2(b).
Besides, for positive instances X p and other pseudo-negative instances
X u \ X mpn , we uniformly choose their mixup partners from X p ∪ X u .
The overall mixup partner selection is formulated as follows:
[Equation]
In what follows, we introduce how to estimate the marginal pseudo-negative instances X mpn and construct the candidate mixup pool X cnd .
Marginal pseudo-negative instance estimation.
Because the fully supervised boundary is exactly unknown, we have to estimate the set of marginal pseudo-negative instances X mpn from X u .
In this work, we deﬁne them as the “unreliable” pseudo-negative instances measured by the predictive scores with thresholding parameter γ ∈ [0 .
[Equation]
where γ = 0 .
5 implies X mpn
= ∅ , and γ = 1 means X mpn
= X u .
Candidate mixup pool.
We maintain a candidate mixup pool X cnd containing the positive instances around the current learned boundary from P .
To be speciﬁc, for each positive instance we compute its entropy value of the predictive score, and update the candidate mixup pool with the top- k positive instances as follows:
[Equation]
where H ( · ) is the entropy, and Rank ( · ) outputs a set of positive instances with the top- k maximum entropy values.
For efﬁciency, we update X cnd per-epoch.
The full training procedure is shown in
Algorithm 1 .
2. R OBUSTNESS
The augmented instances within (cid:98) X u also suffer from imprecise supervision even using the heuristic mixup.
To make P 3 M ix more robust, we employ two tricks, i.e., early-learning regularization (Liu et al., 2020) and pseudo-negative instance correction.
We call the versions with early-learning regularization and pseudo-negative instance correction as P 3 M ix- E and P 3 M ix- C , respectively.
Early-learning regularization.
We employ the early learning regularization to prevent the memorization of imprecise supervision (Liu et al., 2020).
For each mixup instance within (cid:98) X p ∪ (cid:98) X u , we estimate an auxiliary target vector (cid:101) y , and formulate the early-learning regularization below:
[Equation]
Here, we estimate the target vector (cid:101) y of each instance by using the mean teacher technique (Tarvainen & Valpola, 2017), and incorporate Eq.(8) to Eq.(2).
Pseudo-negative instance correction.
We concentrate on the pseudo-negative instances with high conﬁdence to be positive (cid:8) ( x , y = 0) | ( x , y = 0) ∈
X u , f ( x ; Θ ) > γ (cid:9) .
We directly revise their labels to positive before their corresponding mixup operators.
E XPERIMENT 3. E XPERIMENTAL S ETTINGS
Datasets.
In the experiments, we employ three prevalent benchmark datasets, including FashionMNIST (F-MNIST)
(Xiao et al., 2017), 3 CIFAR-10 (Krizhevsky, 2016), 4 and STL-10 (Coates et al., 2011).
The dataset statistics are described in Table 1.
Note that all benchmark datasets havecategory labels, and we denote them with integers ranging from 0 to 9 following the default settings in torchvision 0.10.0.
For each dataset, we group those category labels into two disjoint sets as positive or negative, and generate two synthetic PU datasets by reversing the deﬁnitions of positive and negative labels.
Following the protocol of (Chen et al., 2020a), the speciﬁc deﬁnitions of labels (“positive” vs “negative”) are as follows: F-MNIST-1: “1,4,7” vs “0,2,3,5,6,8,9”, F-MNIST-2: “0,2,3,5,6,8,9” vs “1,4,7”; CIFAR-10-1: “0,1,8,9” vs “2,3,4,5,6,7”, CIFAR-10-2: “2,3,4,5,6,7” vs “0,1,8,9”; STL-10-1: “0,2,3,8,9” vs “1,4,5,6,7”, STL-10-2: “1,4,5,6,7” vs “0,2,3,8,9”.
For each dataset, we randomly select 1,000 positive instances from the training set, and 500 instances as the validation set.
[Table] [Table]
Baseline methods.
To verify the effectiveness of P 3 M ix, we utilize eight PU learning baselines, including uPU (du Plessis et al., 2014), nnPU (Kiryo et al., 2017), nnPU+mixup, Self-PU
(Chen et al., 2020b), PAN (Hu et al., 2021), VPU (Chen et al., 2020a), M IX PUL (Wei et al., 2020) and
PULNS (Luo et al., 2021), as well as the supervised method for comparison.
The corresponding implementation details of baselines are present in Appendix A. For all comparing methods, we adopt the classiﬁers (including the discriminator of PAN) by LeNet-5 for F-MNIST, and 7-layer CNN for
CIFAR-10 and STL-10.
Specially, the baseline methods of uPU, nnPU and Self-PU require the prior knowledge of class proportion, however, the prior is actually unknown for STL-10 since it contains many “real” unlabeled instances.
Accordingly, we estimate the class proportion of STL-10 by using the SOTA KM2 method (Ramaswamy et al., 2016) before evaluating uPU, nnPU, and Self-PU.
Implementation details of P 3 M ix.
We implement P 3 M ix, P 3 M ix-
E and P 3 M ix- C
by using Pytorch (Paszke et al., 2019) with the Adam algorithm (Kingma & Ba, 2014).
We employ the cross entropy function as the loss function (cid:96) of Eq.(2), ﬁx the mixup hyperparameter α to 1 and the size k of the candidate mixup pool X cnd to 100, and choose the coefﬁcient parameter β from { 0 . 8 , 0 . 9 , 1 . 0 } , the thresholding parameter γ from { 0 . 85 , 0 . 9 , 0 . 95 } .
We will make the sensitivity analysis on { β, γ } later.
Specially, the early-learning regularization parameter of P 3 M ix-
E is chosen from { 1 . 0 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 } .
3. C LASSIFICATION PERFORMANCE
For each dataset, we independently run each comparing method 5 times and report the average classiﬁcation accuracy in Table 2.
Generally,
our P 3 M ix- E and P 3 M ix-
C consistently outperform all PU learning baselines on all benchmark datasets, indicating their superior performance.
Compared with nnPU+mixup, VPU and M IX PUL, which utilize the typical mixup technique, both P 3 M ix-
E
and
P 3 M ix- C achieve signiﬁcant performance gain in most cases, i.e., about 1% ∼ 5% improvements.
These results imply that our proposed heuristic mixup beneﬁts to the supervision correction within marginal pseudo-negative instances.
Compared with the three cost-sensitive PU learning methods uPU, nnPU and Self-PU, P 3 M ix-
E and P 3 M ix- C also outperform them by about 1% ∼ 4% in all cases.
Besides, we observe that all discriminative PU learning methods except uPU perform better than the GAN-based PU learning method PAN in most cases.
The possible reason is that the imprecise supervision within unlabeled instances makes the identiﬁcation of fake instances for the discriminator more difﬁcult, resulting in a worse classiﬁer.
[Table]
[Figure]
C
[Table]
3. A BLATION S TUDY
To evaluate the effectiveness of our proposed heuristic mixup, we conduct the ablation experiments on all benchmark datasets.
Speciﬁcally, we compare P 3 M ix, P 3 M ix- E and P 3 M ix- C with the Disambiguation-Free (DF) objective of Eq.(1), DF+mixup, augmented by the typical mixup technique, and their versions with early-learning regularization (“- E ”) and pseudo-negative instance correction (“- C ”).
The experimental results are reported in Table 3.
It clearly demonstrates that the proposed heuristic mixup can signiﬁcantly improve the classiﬁcation performance.
This result is expected because the heuristic mixup can simultaneously achieve data augmentation and supervision correction by reﬁning the imprecise supervision within marginal pseudo-negative instances.
Besides, we can also observe that both early-learning regularization and pseudo-negative instance correction contribute to the improvement of the classiﬁcation performance in all cases, proving the effectiveness of those two tricks in improving the robustness of models.
3. S ENSITIVITY A NALYSIS
In this section, we examine the sensitivities of the coefﬁcient parameter β and the thresholding parameter γ .
Sensitivity of β .
We examine the impact of different β values over the set { 0 . 1 , 0 . 2 , · · · , 1 . 0 } by
P 3 M ix- C and plot the experimental results in Fig.3.
We omit the results of P 3 M ix and P 3 M ix- E due to their similar performance curves and also page-limitation.
Obviously, the performance achieves the highest and is relatively stable when β ≥ 0 . 8 , and it sharply drops as the values become smaller especially on CIFAR-10-1, CIFAR-10-2, STL-10-1 and STL-10-2.
Notice that the coefﬁcient parameter β is used to balance the importance of the positive and pseudo-negative parts in Eq.(2).
The larger or smaller value of β will result in the indecent importance of the pseudo-negative part, leading to a unstable classiﬁer.
Therefore, we suggest tuning β over the set { 0 . 8 , 0 .
9 , 1 . 0 } in practice.
Sensitivity of γ .
Generally, the thresholding parameter γ is utilized to estimate the set of marginal pseudo-negative instances
X mpn from pseudo-negative instances X u .
However, both the earlylearning regularization and pseudo-negative instance correction may be affected by γ since they are mainly used to control the corrupted negative instances within X u \ X mpn .
Accordingly, we perform the sensitivity analysis of γ by P 3 M ix.
Speciﬁcally, we vary the value of γ over the set { 0 . 5 , 0 .
As shown in Fig.4, P 3 M ix achieves the best performance when γ ∈ { 0 . 85 , 0 .
95 } , and performs relatively worse when γ is too small or too large.
This result is expected because the smaller value of γ implies that X mpn contains fewer marginal pseudo-negative instances, e.g., X mpn
= ∅ when γ = 0 .
5 , and it will hurt the supervision correction of heuristic mixup.
Besides, when γ becomes too large, too many pseudo-negative instances will be treated as marginal pseudo-negative ones, e.g., X mpn =
X u when γ = 1 . 0 , resulting in the reduction of the variety of augmented instances.
In summary, its suggested setting is given by { 0 . 85 , 0 . 9 , 0 . 95 } .
R ELATED W ORK
In this section, we review the representative studies on PU learning, especially the ones most related to P 3 M ix.
Besides, we brieﬂy introduce the recent studies of mixup for data augmentation.
4. PU L EARNING
Weakly supervised learning (Zhou, 2018) mainly tackles datasets with weak supervision, such as incomplete labels (van Engelen & Hoos, 2020; Li et al., 2021a), inexact labels (Feng et al., 2020a;b;
Li & Wang, 2020; Li et al., 2020a; 2021b), and inaccurate labels (Li et al., 2020b; Nguyen et al., 2020).
PU learning is an emerging paradigm of weakly supervised learning.
The early PU learning works focus on the sample-selection paradigm.
As the name suggests, the basic idea of sampleselection methods is to select reliable negative instances from unlabeled instances to form pseudobinary dataset before applying supervised methods.
Existing methods have proposed various heuristic strategies for negative sample selection, e.g., 1-DNF (Yu et al., 2002; 2004; Peng et al., 2008),
Na¨ıve Bayes (Liu et al., 2002), Rocchio extraction (Li & Liu, 2003), k NN (Zhang & Zuo, 2009), k -means (Chaudhari & Shevade, 2012), large margin method (Gong et al., 2018) and reinforcement learning (Luo et al., 2021).
Early works based on sample-selection spirit mainly focus on exploiting various traditional classiﬁcation and clustering approaches to construct the heuristic strategy.
Recently, PULNS (Luo et al., 2021) employs a negative selector trained by a reinforcement learning framework, where the selector, the selection of negative instances, and the performance of the classiﬁer are treated as the agent, action and reward, respectively.
Then the classiﬁer is induced from the mixture of positive instances and negative ones selected by the selector.
In contrast, the community of PU learning has recently paid more attention to the cost-sensitive methods, which directly treat all unlabeled instances as corrupted negative instances and correct the estimation bias of the objective by employing well-designed misclassiﬁcation risks.
The uPU (du Plessis et al., 2014; 2015) made an early attempt of unbiased risk estimation, which reformulates the misclassiﬁcation risk as an equivalent and ubiased form depending only on PU datasets.
However, as reported in (Kiryo et al., 2017), the risk of uPU would become negative due to overﬁtting when using the ﬂexible and complex models such as deep networks.
To remedy this issue, the authors of (Kiryo et al., 2017) suggest the nnPU method, i.e., the non-negative version of uPU.
Besides, the Self-PU (Chen et al., 2020b) further considers the learning capability of the model itself, and jointly employs three self-supervision techniques, i.e., a self-paced strategy to discover conﬁdent positive and negative instances, a self-calibrated instance-aware loss to explore meaningful supervision over unconﬁdent instances, and a self-supervision consistency by teacher-students learning.
Other cost-sensitive methods are based on the maximum margin objective, and reﬁne the bias of corrupted negative instances by various tricks, such as unbiased centroid estimation of unlabeled instances (Shi et al., 2018;
Gong et al., 2019b), label calibration with a hat loss (Gong et al., 2019a), and margin-based label disambiguation (Zhang et al., 2019).
In parallel with the aforementioned methods, several GAN-based PU learning methods (Hou et al., 2018; Chiaroni et al., 2018;
Guo et al., 2020; Na et al., 2020;
Hu et al., 2021) have been proposed.
The GenPU (Hou et al., 2018) generates positive and negative instances, and induces a classiﬁer from those generated instances.
The PAN (Hu et al., 2021) is based on adversarial learning on the probability distributions of a discriminator and a classiﬁer.
Orthogonal to those PU learning methods, our P 3 M ix concentrates on data augmentation and extends the well-established mixup technique (Zhang et al., 2018) to a speciﬁc heuristic version for PU learning, enabling to achieve data augmentation and supervision correction simultaneously.
The recent PU learning works most related to P 3 M ix are VPU (Chen et al., 2020a) and MixPUL (Wei et al., 2020), in which the mixup is used as a regularization to improve the robustness of classiﬁers and performed among labeled and unlabeled instances randomly.
In contrast, our P 3 M ix constructs a heuristic mixup partner selection to reﬁne the imprecise supervision within unlabeled instances.
4. M IXUP A UGMENTATION
The mixup technique (Zhang et al., 2018) generates augmented instances with convex combinations of training instances.
Despite its simplicity, it can effectively improve the robustness with even scarce and noisy supervision (Thulasidasan et al., 2019; Carratino et al., 2020; Zhang et al., 2021).
Further, a number of modiﬁed mixup versions (Verma et al., 2019; Yun et al., 2019; Guo et al., 2019;
Kim et al., 2020; Hendrycks et al., 2020) have been proposed, e.g., manifold mixup that generates convex combinations on the latent feature space (Verma et al., 2019) and puzzle mixup that further explores the saliency information and underlying statistics of instances (Kim et al., 2020).
In this work, we extend the typical mixup to a speciﬁc heuristic version for PU learning.
C ONCLUSION
In this paper, we propose a novel PU learning method named P 3 M ix, which extends the typical mixup technique to a heuristic version.
The story begins with the observation of the decision boundary deviation phenomenon, which inspires us to propose a guidance of mixup partners selection, especially for the marginal pseudo-negative instances.
Fortunately, this heuristic mixup technique can simultaneously achieve data augmentation and supervision correction for PU learning.
Generally, our P 3 M ix is easy-to-implement, and we also employ two tricks to improve the robustness of P 3 M ix.
We compare P 3 M ix against a number of existing PU learning methods on benchmark datasets.
Experimental results show the superior performance of P 3 M ix and the effectiveness of the heuristic mixup technique
A CKNOWLEDGMENTS
We would like to acknowledge support for this project from the National Key R&D Program of
China (No.2021ZD0112501, No.2021ZD0112502), the National Natural Science Foundation of
China (NSFC) (No.61876071, No.62006094), the Key R&D Projects of Science and Technology
Department of Jilin Province of China (No.20180201003SF, No.20190701031GH).
A D ETAILS OF B ASELINE M ETHODS
Eight existing PU learning baselines and the supervised method are employed for comparison in this paper.
The details of baseline methods are presented below.
• u nbiased PU learning ( uPU ) (du Plessis et al., 2014):
A cost-sensitive method based on unbiased risk estimation.
We use the public code from the net.• n on- n egative PU learning ( nnPU ) (Kiryo et al., 2017):
A cost-sensitive method based on non-negative risk estimation.
We use the public code from the net.
6 [suggested settings: β = 0 and γ = 1 . 0 ] • nnPU+mixup : A cost-sensitive method incorporating the typical mixup technique into the nnPU method by mixing positive instances and unlabeled ones separately.
• Self-PU
(Chen et al., 2020b):
A cost-sensitive method with self-supervision scheme.
We use the public code from the net.
7 [suggest settings: α = 10 . 0 , β = 0 .
3 , γ = 1 / 16 , Pace
= 0 .
2 and Pace2 = 0 .
3 ] • P redictive
A dversarial N etworks ( PAN ) (Hu et al., 2021):
A GAN-based PU learning method with a discriminator and a classiﬁer.
We use the public code from the net.
8 [suggested settings: λ = 1 e − 4 ] • V ariational PU learning ( VPU ) (Chen et al., 2020a):
A PU learning method based on the variational principle.
We use the public code from the net.
9 [suggested settings: α = 0 .
3 , β ∈ { 1 e − 4 , 3 e − 4 , 1 e − 3 , · · · , 1 , 3 } ]
• M IX PUL (Wei et al., 2020):
A PU learning method based on the consistency regularization with the mixup technique.
We use the public code from the net.
10 [suggested settings: α = 1 . 0 , β = 1 . 0 , η = 1 . 0 ]
• P
ositive-
U nlabeled L earning with effective N egative sample S elector ( PULNS )
(Luo et al., 2021):
A sample-selection method with reinforcement learning.
We implement an in-house python code with a 3-layer MLP selector suggested by the paper.
[suggested settings: α = 1 . 0 and β ∈ { 0 . 4 , 0 .
8 , 1 . 0 } ] • Supervised : The classiﬁers trained on the fully supervised datasets.
[Table]
B A DDITIONAL E XPERIMENTAL R ESULTS ON R EALWORLD D ATASET
To examine the performance of our proposed P 3 M ix- E and P 3 M ix- C in practice
[Table]
we perform the experiments on the Credit Card Fraud Detection task of Kaggle 11 .
We utilize a subset of the original
Credit Card Fraud Detection dataset, which contains all (492) fraudulent instances and 10000 genuine ones selected randomly from the original dataset.
In this subset, the proportion of the positive instances (frauds) is about 0.0469.
We use 20% of the constructed subset as the test dataset
[Table]
and all others as the training one of PU learning, in which 100 frauds are selected randomly as positive instances.
The accuracy, precision and recall are utilized as metrics.
Table 4 reports the average results of independently running 5 times of all comparison methods.
Overall, our proposed P 3 M ix- E
and
P 3 M ix- C gain the best recall score, and also achieve the competitive performance on the accuracy and precision scores, even the dataset is highly unbalanced.