Positive-Unlabeled Learning using Random Forests via Recursive Greedy Risk Minimization
Abstract
The need to learn from positive and unlabeled data, or PU learning, arises in many applications and has attracted increasing interest.
While random forests are known to perform well on many tasks with positive and negative data, recent
PU algorithms are generally based on deep neural networks, and the potential of tree-based PU learning is under-explored.
In this paper, we propose new random forest algorithms for PU-learning.
Key to our approach is a new interpretation of decision tree algorithms for positive and negative data as recursive greedy risk minimization algorithms .
We extend this perspective to the PU setting to develop new decision tree learning algorithms that directly minimizes PU-data based estimators for the expected risk.
This allows us to develop an efficient PU random forest algorithm, PU extra trees.
Our approach features three desirable properties: it is robust to the choice of the loss function in the sense that various loss functions lead to the same decision trees; it requires little hyperparameter tuning as compared to neural network based PU learning; it supports a feature importance that directly measures a feature’s contribution to risk minimization.
Our algorithms demonstrate strong performance on several datasets.
Our code is available at https://github.com/puetpaper/PUExtraTrees .
Introduction
Positive and unlabeled learning (PU learning) has attracted increasing interest recently, due to its broad applications, including disease gene identification [24], landslide susceptibility modeling [29], inference of transcriptional regulatory network [23], cybersecurity [30] and many others [2].
Existing algorithms generally reduce PU learning to solving one or more supervised learning problems.
A naive approach is to simply learn a classifier by treating all unlabeled examples as negative, but unlabeled examples can be either positive or negative.
Various approaches have been developed to better handle the uncertainty on the labels of the unlabeled examples.
A common approach follows an iterative two-step learning process: the first step identifies reliable negative examples, then the second step learns a model using the positive examples, the reliable negative examples and possibly unlabeled examples [22, 15].
Another common approach transforms the PU dataset into a weighted fully labeled dataset [20, 12, 11, 16].
The transformed dataset is often constructed so that the empirical risk on the transformed dataset provides a good estimate of the true risk.
Depending on the assumptions on how
PU data is generated, different transformations have been developed.
While random forests are known to perform well for many supervised learning problems and in principle tree-based learning methods can be applied in the above approaches, little work has been done to explore the potential of tree-based methods for PU learning.
On the other hand, current state-of-the-art PU learning methods are often based on deep neural networks [16, 6].
This is partly due to the recent advances of deep learning, which allows training of powerful neural network models with relatively little effort.
In addition, while it is easy to train a neural network to minimize different loss functions, it is not clear how this can be done for tree-based methods (without overfitting the dataset), because existing tree-based methods have not been designed to directly minimize the loss.
In particular, it is not clear how this can be done for nonstandard risk estimators such as the non-negative
PU risk (nnPU), which has already been observed to work well with neural networks [16].
We fill this gap by providing a theoretically justified approach, which we call recursive greedy risk minimization , to learn random forests from PU data in this paper.
Our contributions are as follows
• A new interpretation to existing decision tree learning algorithm that links the impurity reduction based learning algorithms to empirical risk minimization.
• Efficient new decision tree and random forest algorithms that directly minimize PU-data based estimators for the expected risk.
• Our approach has three desirable properties: it is robust to the choice of the loss function in the sense that various loss functions lead to the same decision trees; it requires little hyperparameter tuning as compared to neural network based PU learning; it supports a feature importance that directly measures a feature’s contribution to risk minimization.
• Our algorithms demonstrate strong performance on several datasets.
We briefly review decision tree learning and PU learning in Section 2, introduce our recursive greedy risk minimization approach for decision tree learning in Section 3 and the PU version of extra trees in
Section 4, present experiment results in Section 5, discuss related works in Section 6, and conclude in Section 7.
Background
Decision tree learning from positive and negative data A decision tree is generally constructed by recursively partitioning the training set so that we can quickly obtain subsets which are more or less in the same class.
1 Notations: κ - node; S - dataset; f - feature; tthreshold; 2 if termination criterion is met then
Compute else f>t ) ;
LearnDT ( κ f ≤ t , S f ≤ t ) ; 9 end
Algorithm 1 shows a generic decision tree learning algorithm.
We start with a single node associated with the entire set of training examples.
When we encounter a node κ associated with a set S of training examples, we compute the prediction value at κ if the termination criterion is met.
If not, we first compute an optimal split ( f, t ) for the dataset S , where a split ( f, t ) tests whether a feature f is larger than a value t , and the quality of a split is usually measured by its impurity reduction.
Based on the test’s outcome, we create two child nodes κ f>t and κ f ≤ t , split S into two subsets S f>t and S f ≤ t , and finally continue the learning process with ( κ f>t , S f>t ) and ( κ f ≤ t , S f ≤ t ) .
Given an impurity measure Impurity ( S ) , a split ( f, t ) ’s impurity reduction is
[Equation]
Typically, the Gini impurity G ( S ) and the entropy H ( S ) are used as the impurity measure:
[Equation]
where q + and q − are the proportions of positive and negative examples in S respectively.
We will use IR Gini ( f, t ; S ) and IR entropy ( f, t ; S ) to denote the impurity reduction when using the Gini impurity and the entropy impurity respectively.
PU learning We assume that the input x ∈ R d and label y ∈ {− 1 , +1 } follow an unknown joint distribution p ( x , y )
=
p ( x | y ) p ( y ) .
In the fully supervised case, the training data is generally assumed to be independently sampled from p ( x , y ) .
In the PU setting, one common assumption on the data generation mechanism [11, 10, 16] is that the positive examples P = { x ( p ) i } n p i =1 are sampled independently from the P marginal p p ( x ) :
= p ( x | y = 1) , and the unlabeled examples
U = { x ( u ) i } n u i =1 are sampled independently from the marginal p ( x ) =
p ( x , +1) + p ( x , − 1) .
Clearly, we have p ( x ) =
π p p ( x ) + (1 − π ) p n ( x ) , where π := p ( y = +1) is the positive rate and p n ( x ) :
= p ( x | y = − 1) is the N marginal.
The objective is to learn a score function g : R d → R (which corresponds to a classifier that outputs +1 iff g ( x ) > 0 ) such that it minimizes its expected risk
[Equation]
where ℓ : R × {− 1 , +1 } → R is a loss function that gives the loss ℓ ( v, y ) incurred by predicting a score v when the true label is y .
We focus on the quadratic loss ℓ quad ( v, y ) =
(1 − vy ) 2 , the logistic loss ℓ logistic ( v, y ) =
ln(1 + exp( − vy )) , the savage loss ℓ savage ( v, y )
= 4 / (1 + exp( vy )) 2 and the sigmoid loss ℓ sigmoid ( v, y )
= 1 / (1 + exp( vy )) in this paper.
The risk can be written in terms of positive and unlabeled data as [11]
[Equation]
This gives the following unbiased PU-data based risk estimator
[Equation]
where w p = π/n p and w u = 1 /n u .
While the uPU risk estimator is unbiased, it has a negative component that provides extra incentive for the classifier to try hard to fit to the positive examples, thus potentially making the risk negative and leading to overfitting.
The non-negative risk estimator
[Equation]
alleviates this problem by forcing the sum of the negative term and the term defined on the unlabeled data to be non-negative, as the sum acts as an estimate for risk on negative examples
Recursive Greedy Risk Minimization 3.1 Decision trees for positive and negative data
We first introduce our recursive greedy risk minimization approach in the fully labeled case.
Consider the empirical risk of g : R d → R on a labeled training set D
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } : (cid:98) R ( g ) =
(cid:80) ( x
,y ) ∈
D wℓ ( g ( x ) , y ) , where w = 1 / | D | .
If g predicts a constant v on a subset S = { ( x , y ) } of the training examples, then the contribution to the total empirical risk is the partial empirical risk (cid:98) R ( v ; S ) :=
[Equation]
( x ,y ) ∈ S wℓ ( v, y ) .
The optimal constant value prediction is v ∗ S = argmin v ∈ R (cid:98) R ∗ ( v ; S ) with a minimum partial empirical risk (cid:98) R ∗ ( S ) :
= (cid:98) R ∗ ( v ∗ S ; S ) .
If we switch from a constant value prediction rule to a decision stump that uses the split ( f, t ) , then the minimum partial empirical risk for such a decision stump is (cid:98) R ∗ ( S f>t )
+ (cid:98) R ∗ ( S f ≤ t ) , thus the risk reduction for the split ( f, t ) is
[Equation]
In the following, when we need to make the loss function explicit, we will use subscript to indicate that.
For example, (cid:98) R ∗ quad ( S ) and RR quad ( f, t ; S ) , are the minimum partial empirical risk for a constant-valued prediction on S and the risk reduction for the split ( f, t ) on S when using the quadratic loss.
When we choose the ( f, t ) with maximum risk reduction in Algorithm 1, we obtain a recursive greedy risk minimization algorithm that recursively zooms in to an input region with a constant prediction, then improves the prediction rule in a greedy manner by replacing it with the decision stump with minimal empirical risk in that region.
The first main result shows that the Gini impurity reduction and entropy reduction of a split are just scaled versions of the risk reductions when using the quadratic loss and the logistic loss respectively.
This implies that the standard impurity reduction based decision learning algorithm is in fact performing recursive greedy risk minimization.
All proofs are in the appendix.
Theorem
(a) For any S ⊆ D , we have (cid:98) R ∗ quad ( S )
= 2 | S | wG ( S ) .
As a consequence, for any S ⊆ D and any split ( f, t ) [4] ,
[Equation]
(b) For any S ⊆ D , we have (cid:98) R ∗ logistic ( S ) =
| S | wH ( S ) .
As a consequence, for any S ⊆ D and any split ( f, t ) ,
[Equation]
3.2 Decision trees for positive and unlabeled data
We are now ready to introduce a recursive greedy risk minimization approach to PU learning.
We do this by making two changes to Algorithm 1: the fully labeled dataset S is replaced by a set P ′ ⊆
P of positive examples and a set U ′ ⊆
U of unlabeled examples, and the split ( f, t ) is chosen to optimize a PU version of the risk reduction.
uPU risk reduction We first consider the simpler case of the uPU estimator.
Similarly to the fully labeled case, if g predicts a constant v on P ′ and U ′ , then the contribution to the total empirical uPU risk is
[Equation]
u ℓ ( v, − 1) .
The optimal constant value prediction is v ∗ P ′ ,U ′ =
argmin v ∈ R (cid:98) R ∗ uPU ( v ; P ′ , U ′ ) with a minimum empirical risk (cid:98) R ∗ uPU ( P ′ , U ′ ) :
= (cid:98) R ∗ uPU ( v ∗ P ′ ,U ′ ; P ′ , U ′ ) .
The uPU risk reduction for the split ( f, t ) on ( P ′ , U ′ ) is
[Equation]
The following result shows that the optimal risk (cid:98) R ∗ uPU ( P ′ , U ′ ) has a closed-form formula and is thus efficiently computable.
As a consequence, the uPU risk reduction for a split ( f, t ) is efficiently computable.
Proposition 1.
Consider arbitrary P ′ ⊆
P and U ′ ⊆
U ′ .
Let W p = | P ′ | w p , W n = | U ′ | w u −| P ′ | w p , and v ∗ :=
W p W p + W n .
(a) When using the quadratic loss, we have
[Equation]
(b) When using the logistic loss, we have
[Equation]
Intuitively, if P ′ ⊆
P and U ′ ⊆
U are the examples in a node κ in the decision tree, then the weights
W p and W n serve as unbiased estimates of the probabilities p ( x ∈ κ, y = 1) and p ( x ∈ κ, y = − 1) respectively.
See Appendix C for details.
Thus v ∗ =
W p W p + W n serves as an estimate of the probability of positive examples.
Note that W p ∈ [0 , 1] , while W n ≤ 1 is unusual in that it contains a negative component and hence can be negative.
Consequently, v ∗ is non-negative, but it can be larger thanthough.
In particular, if W p + W n = | U ′ | w u = 0 , then v ∗ = + ∞ .
nnPU risk reduction For the case of the nnPU estimator, define
[Equation]
We can then similarly define the risk reduction for the nnPU estimator.
The following result shows that in the nnPU case, the optimal risk (cid:98) R ∗ uPU ( P ′ , U ′ ) also has a closedform formula and is thus efficiently computable.
Proposition 2.
Using the same notations as in Proposition 1, we have the following results.
(a) When using the quadratic loss, we have
[Equation]
(b) When using the logistic loss, we have
[Equation]
It is important to note that while in the uPU case, if P ′ is partitioned into P ′ 1 and P ′ 2 , and U ′ into
U ′ 1 and U ′ 2 , then we have (cid:98) R uPU ( u ; P ′ , U ′ ) =
[Equation]
R
[Equation]
u, P
[Equation] [Equation]
U
[Equation]
1 ) + (cid:98) R uPU ( u, P ′ 2 , U ′ 2 ) , we only have
[Equation]
R nnPU
[Equation]
u
[Equation]
P
[Equation]
U
[Equation]
R
[Equation]
R nnPU ( u, P ′ 2 , U ′ 2 ) .
Thus recursive greedy risk minimization minimizes an upper bound of the partial empirical risk rather than the partial empirical risk.
This can be considered as a regularization mechanism.
In fact, we observe nnPU risk to work better than the uPU risk in our experiments, which is consistent with the findings in [16].
Optimizing the split With the above closed-form formula, we can then efficiently compute the risk reduction for a given split ( f, t ) on ( P ′ , U ′ ) .
While there are infinitely many possible t values to choose from, note that they often split the dataset in the same way, and if f ’s values split R into multiple intervals, we only need to consider t 1 < t 2 < . . .
< t m , each chosen from an interval so that we cover all possible cases.
While computing the risk reduction for each ( f, t i ) takes O ( | P ′ | + | U ′ | ) time, we can efficiently get the risk reductions for all of ( f, t 1 ) , . . .
, ( f, t m ) in
O (( | P ′ | + | U ′ | )
ln( | P ′ | + | U ′ | ) + m ) time, by sorting the examples according to their f values, and going through the thresholds in a sorted order.
See Appendix E for details.
Invariance properties An interesting property of the greedy recursive learning approach is that it is robust to the choice of the loss function in the sense that different loss functions can lead to identical decision trees, as shown in the result below.
Proposition 3.
The (cid:98) R ∗ ( S ) value is the same for the quadratic loss and the savage loss.
Another interesting property is that the optimal prediction among − 1 and +1 are often the same when we use different estimators and different loss functions.
Proposition 4.
If a node contains the examples P ′ ⊆
P and U ′ ⊆
U , v ∗ is as defined in Proposition 1, and the prediction is chosen from {− 1 , +1 } to minimize the empirical risk estimate on P ′ and U ′ , then the optimal prediction is 2 1 ( v ∗ > 0 .
5) − 1 for both the uPU and nnPU risk estimate and all the loss functions in Section 2.
Intuitively, we predict +
iff the estimated proportion of positive data at the current node is larger than 0.5.
PU Extra Trees
With our PU decision tree algorithms, we can obtain a PU random forest algorithm by adapting the random forest algorithm [3] for positive and negative data: repeatedly obtain bootstrap samples for both the positive data and the unlabeled data, then train a PU decision tree for each sample with optimal split chosen over a randomly sampled subset of features each time.
While choosing the optimal split from a subset of features makes training each decision tree faster, it is still the most expensive part in the tree construction process and can potentially be very slow on a large dataset.
We describe a more efficient version that we used in our experiments below.
In addition, we introduce a new feature importance score that directly measures a feature’s contribution to risk minimization.
PU ET (Extra Trees)
We develop a more efficient random forest algorithm by using the randomization trick in the Extra Trees algorithm [13] to further reduce the computational cost for finding an optimal split below.
Besides sampling only a subset of features, only one random threshold for each sampled feature is considered.
We implemented a more general version which allows sampling multiple random thresholds for a sampled feature.
A complete pseudocode for PU ET is given in
Appendix H.
Termination criteria Decision trees can overfit the training data if allowed to grow until each leaf has just one example.
Learning is generally terminated early to alleviate overfitting.
In our implementation, we terminate when all feature values are constant, when a maximum tree depth d max is reached, when the node size falls below a threshold s min , or when the node is pure .
A node is said to be pure if the impurity measure takes value −∞ in the uPU setting, or value 0 in the nnPU setting.
Risk reduction importance Our feature importance, called risk reduction importance , is the total risk reduction for a feature across all nodes in all the trees.
The risk reduction importance of a feature f is defined as (cid:80) κ ∈ K f RR( f κ , t κ ; P κ , U κ ) , where K f is the set of nodes using f as a splitting feature, ( f κ , t κ ) is the split for node κ , and ( P κ , U κ ) is the set of PU data at κ .
This can be averaged across multiple trees in a random forest.
The risk reduction importance in PU learning is similar to the Gini importance as defined in, for example, [4], whereby the size of the nodes is taken into account.
This is beneficial as otherwise features often used in splitting small datasets may appear very important as compared to features used in splitting large datasets, even though it does not contribute as much to reducing the risk and improving classification performance.
To demonstrate the effect of taking data size into account, a normalized risk reduction can be defined by dividing the risk reduction by the total weight of the examples.
Experiments
In this section we compare PU Extra Trees with several other PU learning methods.
Selected treebased methods include NaivePUET whereby ET is trained on the PU dataset by simply treating
U data as N data; PUBaggingET which uses PUBagging from [23] with ET base classifier; and
SupervisedET where ET is trained on the original fully labeled dataset (expected to be an upper bound for tree based PU methods).
To further study the effectiveness of our PU Extra Trees algorithm we also compare against neural network based methods including a baseline method uPU [11, 10] and two state of the art methods nnPU [16] and Self-PU [6].
We ran experiments on heterogeneous devices.
In particular, random forests were trained using 32GB RAM and one of Intel i7-10700,
Intel i7-11700 or AMD Epyc 7702p CPU.
Neural networks were trained on one of NVIDIA RTX
A4000 or NVIDIA RTX A6000 GPU
due to the lack of identical devices.
While the running times are not always comparable and thus not provided in the main text, experiments show that our current implementation of PU ET takes seconds to train a single-tree random forest on modest hardware.
See
Appendix I for details.
Datasets We consider a selection of common datasets for classification from LIBSVM [5], as well as MNIST digits [19], the intrusion detection dataset UNSW-NB15 [25] and CIFAR-10 [17] to demonstrate the versatility of our method.
Table 1 is a summary of the benchmark datasets.
[Table]
The 20News, epsilon, MNIST and CIFAR-10 datasets were processed in the same way as in [16] for consistency.
Definitions of labels (‘positive’ vs ‘negative’) are as follows:
For 20News, ‘alt., comp., misc.
, rec.’
vs ‘sci., soc.
, talk.’
; for MNIST, ‘0,2,4,6,8’ vs ‘1,3,5,7,9’; for CIFAR-10, ‘airplane, automobile, ship, truck’ vs ‘bird, cat, deer, dog, frog, horse’; for UNSW-NB15, all attack types make up the P class and the benign data makes up the N class; for mushroom and covtype, the most prominent class makes up the P class; and epsilon dataset is provided with P and N classes.
GloVe pre-trained word embeddings [27] were used for 20News with average pooling over each document.
Each feature in the covtype and UNSW-NB15 datasets was scaled between 0 and 1 when using uPU and nnPU as there is a large mismatch in scales between the features which seemed to significantly reduce performance of the methods.
No such scaling of the data is required for PU ET as performance does not seem to be affected in such cases.
To convert the PN data to PU data we follow the experimental setup in [16] whereby 1000 positive examples are randomly sampled to form the P set, and the entire dataset is used for the U set.
In practice it may be the case that π is not known a-priori, in which case there are methods for estimating π using only PU data, for example, in [7], or [28].
Model hyperparameters Following common practice [26, 13], the default hyperparameters for
PU ET are: 100 trees, no explicit restriction on the maximum tree depth, sample F = ⌈√ d ⌉ features out of a total of d features and sample T = 1 threshold value when computing an optimal split.
The size of the bootstrap sample for PUBaggingET is set to the default value of the dataset size.
The architectures for the neural networks used in uPU, nnPU and Self-PU were copied from [16] for the 20News, epsilon, MNIST and CIFAR-10 datasets.
A 6 layer MLP with ReLU was used for MNIST, Covtype, Mushroom and UNSW-NB15; a similar model was used for epsilon while the activation was replaced with Softsign; a 5 layer MLP with Softsign was used for 20News.
All hidden layers had width 300 for the MLPs.
The model for CIFAR-10 was thelayer CNN: (32*32*3,1)-[C(3*3,96,1)]*2-C(3*3,96,2)-[C(3*3,192,1)]*2-C(3*3,192,2)-C(3*3,192,1)-
C(
*1,192,1)-C(1*1,10,1)-1000-1000-1, where the input is a 32*32 RGB image, C(3*3,96,1) means 96 channels of 3*3 convolutions with stride 1 followed by ReLU, [.]*2 means there are two such layers.
For the remaining datasets we used a 6-layer MLP with ReLU [18, 14] (more specifically, d -300-300-300-300-1).
For each dataset the neural networks were trained for 200 epochs.
The batch size, learning rate, use of batch-norm (for datasets not included in [16]), weight decay and choice of optimiser were tuned for each dataset.
It is noteworthy that we spend little effort to tune PU ET hyperparameters and simply use the same hyperparameters on all datasets.
In addition, PU ET seems to be robust to hyperparameter choice in that the default selection tends to result in strong predictive performance with decreased training times.
See Appendix J for results from hyperparameter tuning experiment.
On the other hand, neural nets are sensitive to hyperparameter choice, and significant tuning is often needed for each dataset.
5.1 Classification performance
To take into account of the randomness in the learning algorithms, we train each model five times on the same dataset, then report the mean and standard deviation of the accuracy and the F-score.
Results for uPU, nnPU and Self-PU were reproduced to the best of our ability, particularly on the datasets that were included in the original results section of the respective papers.
Effect of loss function and risk estimator on PU ET We first evaluate the effect of different loss functions and risk estimators for PU ET.
The results are shown in Table 2 and Table 3.
PU ET with nonnegative risk estimator seems to perform about as well using either quadratic or logistic loss.
Unbiased risk estimator seems to lead to overfitting, but less so with logistic loss.
Logistic loss leads to better performance than quadratic loss when using unbiased risk estimator.
Note that (11) implies that quadratic loss encourages aggressive splitting until all examples are positive as a loss of −∞ can be achieved in this case, while (12) implies that logistic loss has a less aggressive splitting behavior as a loss of −∞ is achieved when the total weight of positive examples exceeds the total weight of unlabeled examples.
As discussed previously, once we achieve an impurity value of −∞ in uPU, we stop splitting.
The more aggressive splitting behavior of quadratic loss leads to deeper trees that tend to overfit more than the logistic loss in general.
This is consistent with results from additional experiments in Appendix K.
Comparison of
PU ET
with other tree-based PU learning methods We now compare our PU ET algorithm with several tree-based baselines to further study the effectiveness of our novel PU tree
[Table]
[Table]
learning method.
The results are shown in Table 4 and Table 5.
The results support the effectiveness of our algorithm.
Our PU ET with nnPU risk estimator and quadratic loss significantly outperforms both
NaivePUET and PUBaggingET, particularly in terms of F-scores.
PU ET shows strong performance even as compared to SupervisedET, though often using a small proportion of the positive labels only.
[Table]
[Table]
Comparison of PU ET with neural network methods We compare PU ET with nnPU risk estimator and quadratic loss against uPU neural nets, nnPU neural nets and Self-PU.
The results are shown in Table 6 and Table 7.
PU ET shows strong performance in terms of both accuracy and F score on a wide variety of datasets compared to neural network based models
[Table] [Table]
[Table]
5.2 Feature Importance
Interpretability is a desirable property of machine learning algorithm, but neural networks are often hard to interpret.
In contrast, tree based methods are generally easier to interpret.
We illustrate our risk reduction importances of both PU ET and PN (positive-negative) ET on the UNSW-NB15 and the MNIST datasets (lighter color indicates a higher score) in Figure 1 and Figure 2 respectively .
s tt l d tt l c t _ s t a t e _ tt l d m e a n s w i n r a t e d l o a d d w i n c t _ s r v _ d s t c t _ s r v _ s r c
d t c pb s m e a n c t _ d s t _ s r c _ l t m t c p r tt s t c pb c t _ s r c _ l t m c t _ d s t _ l t m c t _ s r c _ dp o r t
_ l t m s y n a c k i s _ s m _ i p s _ p o r t s s i n p k t s l o a d a c k d a t c t _ d s t _ s p o r t _ l t m d u r s b y t
e s t r a n s _ d e p
t h s ji t s p k t s d l o ss s
l
o ss d i n p k t
d ji t dp k t s db y t
e s c t
f l w _ h tt p _ m t
h d
i s _ f t p _
l o g i n c t _ f t p _ c m d r e s p o n s e _ b o d y _ l e n
[Figure]
s tt l d tt l c t _ s t a t e _ tt l d l o a d c t _ s r v _ d s t s w i n c t _ s r v _ s r c r a t e c t _ d s t _ s r c _ l t m s m e a n d m e a n c t _ d s t _ s p o r t
_ l t m s l
o a d d w i n d t c pb a c k d a t s i n p k t s
b y t
e s i s _ s m _
i p s _ p o r t s c t _ s r c _ dp o r t
_ l t m t c p r tt s
t c pb s y n a c k c t
_ s r c _ l t m c t _ d s t _ l t m d u r s ji t s p k t s d ji t d i n p k t s l o ss db y t
e s d l o ss dp k t s t r a n s _ d e p t h c t
_ f l w _ h tt p _ m t h d r e s p o n s e _ b o d y _ l e n i s _ f t p _
l o g i n c t _ f t p _ c m d
[Figure]
For MNIST, the plot of the importance scores for PU ET for each digit often suggests the shape of the digit.
On the other hand, the Appendix provides some additional figures that show that the normalized risk reduction importance makes many more pixels more important.
This observation is consistent with our discussion in Section 4.
Interestingly, the importances for PU ET appear to be very similar to those for PN ET (visually similar importance maps for MNIST, and qualitatively similar importance values which give similar feature rankings for UNSW-NB15).
This suggests that the PU model is likely quite similar to the PN model.
Related Work
PU learning was initially studied in [9, 8, 21], and many approaches have been developed based on different assumptions on the data generation mechanism.
The two-step approach discussed in the introduction have been adopted in many papers with numerous instantiations.
We refer the readers to [2] for a comprehensive list.
Another important line of work, which is most related to our work and has achieved the state-ofthe-art performance recently, minimizes a risk estimate based on a weighted dataset of positive and negative examples obtained by transforming the PU dataset [20, 12, 11, 16].
Our work follows the risk minimization approach, but we develop general risk minimization algorithms for learning tree models, that allows exploiting the under-explored potential of random forests.
Numerous works have been done on tree-based methods, and random forest algorithms are among the most successful ones [3, 13, 1].
In the PU learning literature, PU decision trees were explored in early works [8, 21], and PU bagging [23] has been explored for SVMs [23] and decision trees [29].
All of these approaches do not aim to directly minimize the risk.
Our recursive greedy risk minimization framework gives a new interpretation to existing tree learning algorithms for positive and negative data, and allows us to develop new algorithms for positive and unlabeled data.
In addition, our approach allows us to formulate a new feature importance score that directly measures a feature’s contribution to risk minimization.
Conclusions
Random forests perform very well on fully supervised learning methods, but its potential in PU learning has been under-explored due to the strong emphasis on neural networks and the lack of a principled method for learning decision trees from PU data.
We fill this gap in this paper.
We first consider learning from positive and negative data, and develop a recursive greedy risk minimization approach for learning decision trees, and show that with suitable choice of loss functions, its instantiations are equivalent to standard impurity-based decision tree learning algorithms.
We extend this approach to the PU setting, and develop a very efficient random forest algorithm which requires little hyperparameter and yet strong performance against the state-of-the-art PU learning algorithms.
Our approach supports a new feature importance score that directly measures a feature’s contribution to risk mnimization.
Our work opens up some new opportunities to build tree models.
First, decision trees can easily overfit the training set, and various heuristics have been used in practice to alleviate overfitting.
Our recursive greedy risk minimization framework provides an alternative way to control the size of the learned decision by always expanding the node with the largest risk reduction until the risk reduction is smaller than a threshold or when the tree size is larger than a threshold.
Second, we focused on quadratic loss and logistic loss in our experiments, but we can also study alternative losses such as the hinge loss, or double hinge loss.
Acknowledgments and Disclosure of Funding
The authors would like to thank Dr Miao Xie from AARNet for his valuable suggestions.
This work was funded by The University of Queensland Cyber Security Seed Funding.
[-On the other hand if 0 ≤ v ∗ < 1 , then we have W p ≥ 0 and W n > 0 and the minimizer will be the solution to 
-]
[-(cid:88) ( x,y ) w (1 + e vy ) 
-]
[-Intuitively, we predict +
-]