D EEP P OSITIVE U NLABELED L EARNING WITH A S EQUENTIAL B IAS
Anonymous authors
Paper under double-blind review
A BSTRACT
For many domains, from video stream analytics to human activity recognition, only weakly-labeled datasets are available.
Worse yet, the given labels are often assigned sequentially , resulting in sequential bias .
Current Positive Unlabeled (PU) classiﬁers, a state-of-the-art family of robust semi-supervised methods, are ineffective under sequential bias.
In this work, we propose DeepSPU , the ﬁrst method to address this sequential bias problem.
DeepSPU tackles the two interdependent subproblems of learning both the latent labeling process and the true class likelihoods within one architecture.
We achieve this by developing a novel iterative learning strategy aided by theoretically-justiﬁed cost terms to avoid collapsing into a naive classiﬁer.
Our experimental studies demonstrate that DeepSPU outperforms state-of-the-art methods by over 10% on diverse real-world datasets.
I NTRODUCTION
Motivation.
State-of-the-art approaches for learning from data with only only incomplete positive labels require an accurate estimation of the likelihood that any given positive instance receives a label, known as the propensity score .
However, all existing approaches overlook the fact that the annotations given for sequential data are often clustered together, and thus the likelihood that a given instance is labeled is dependent on the labels of the surrounding instances.
We refer to this as sequential bias .
Overlooking this sequential bias results in an incorrect propensity score and signiﬁcantly reduced classiﬁcation performance.
Ours is the ﬁrst work to make this observation and we propose the ﬁrst solution to this open problem.
Human Activity Recognition (HAR) is a prime example of sequential bias in data.
To collect HAR data, subjects are asked to report their activities while wearing mobile sensors.
As study-length increases (collection may take many days), participants leave many activities unlabeled.
Additionally, wearable sensors record data rapidly so large blocks of time get labeled consecutively, also creating sequential bias.
Many more applications, such as intrusion detection from video or illness prediction from medical records, have similar sequentially-labeled data are are susceptible to sequential bias (Rodr´ıguez-Moreno et al., 2019; Schaekermann et al., 2018).
This is a crucial issue as existing methods show drastically reduced accuracy when sequential bias is not accounted for (as demonstrated in our Experimental Results).
State-of-the-Art.
Positive Unlabeled (PU) classiﬁers are a family of semi-supervised methods that learn from incompletely-labeled data without requiring any labeled negative examples (Bekker &
Davis, 2020; Elkan & Noto, 2008; Li & Liu, 2005; Hsieh et al., 2015; Du Plessis et al., 2015; Kiryo et al., 2017; Bekker & Davis, 2018a; Kato et al., 2019).
This is a key strength of PU methods because representative negative examples, typically required by semi-supervised methods, are often not feasible to acquire.
For instance, in the HAR example, there are inﬁnitely many activities that an individual is not performing at any given time.
Consequentially, participants are only expected to provide some positive labels for their activities (Vaizman et al., 2017).
Unfortunately, existing PU methods make unrealistically restrictive simplifying assumptions on how the labels were applied.
Speciﬁcally, they either assume that there is no bias in the labeling process (the probability of a sample being an unlabeled positive instance is uniform) (Elkan & Noto, 2008;
Du Plessis et al., 2015; Kiryo et al., 2017) or otherwise only depends on the local attributes of each instance (Bekker & Davis, 2018a; Kato et al., 2019).
This means that existing methods do not model sequential bias .
And, as we demonstrate in our experiments, these methods are signiﬁcantly negatively impacted when a sequential bias is present.
Problem Description and Technical Challenges.
Given a dataset of sequences, our goal is to predict the true class likelihoods of each instance within the given sequence given a subset of labeled positive instances during training.
In particular, we focus on the difﬁcult case where labels have been assigned with a sequential bias , with sequential bias deﬁned as the case where the likelihood that a positive instance is labeled varies depending on whether its neighboring instances were labeled.
This problem is challenging due to two difﬁcult interdependent subproblems.
First, we have the dependency problem: if we had a model of the latent labeling process (which we call the propensity model ) that allowed us to identify the true unlabeled positive instances, then we could use this propensity model to train a classiﬁer to produce the true class likelihoods.
However, we need these same true class likelihoods in order to train the propensity model - causing a cyclic dependency.
Second, standard maximum likelihood estimation inherently assumes all instances are labeled, leading to a naive classiﬁer in the presence of labeling bias.
To capture unlabeled positive instances, a PU classiﬁer must instead predict an appropriate number of positive instances without simply assuming all positives instances are labeled.
Our Approach: DeepSPU.
We propose Deep Sequential PU ( DeepSPU ), which is the ﬁrst Positive Unlabeled method to use a propensity score model that predicts the likelihood that any given positive instance is labeled while taking sequential bias into account.
The propensity score allows us to train a classiﬁer network given only partially labeled data.
We achieve this by developing a novel learning method that overcomes the cyclic dependency problem by iteratively learning the propensity score model and the classiﬁer using weakly-labeled data.
Further, we introduce the two novel PU cost terms: the Prior-Matching Costs (PMC) and the Observation-Matching Costs (OMC), which prohibit the propensity model and classiﬁer from collapsing into incorrect naive solutions.
Contributions.
The main contributions of our work are:
• We identify sequential bias , a labeling pattern characteristic of many real-world labeling processes, and demonstrate how ignoring this bias signiﬁcantly impacts the performance of state-ofthe-art PU classiﬁers.
• We propose the ﬁrst learning strategy to minimize the bias incurred from sequentially biased PU data.
Namely, we propose an iterative learning strategy and design two novel PU cost terms, PriorMatching and Observation-Matching, which prohibit collapse into certain incorrect adversarial solutions, as justiﬁed through theoretical analysis.
• We develop DeepSPU, the ﬁrst model to mitigate sequential bias.
DeepSPU uses the aforementioned learning strategy to jointly estimate the two interdependent latent variables: the propensity score and the true class probabilities - without any direct supervision for either learning task.
R ELATED W ORK
PU classiﬁers, a special type of semi-supervised models, learn from weakly-labeled training data without requiring any labeled negative examples (Bekker & Davis, 2020).
In contrast, semisupervised methods lean heavily on having access to both labeled positive and labeled negatives to learn from weakly supervised data, making them less robust than PU alternatives (Van Engelen & Hoos, 2020).
PU methods are also more robust than a similar family of classiﬁers known as One
Class (OC) classiﬁers, as general OC classiﬁers do not make use of unlabeled data during training (Khan & Madden, 2009).
There are many approaches to PU learning, such as re-weighting predictions (Zhang & Lee, 2005;
Elkan & Noto, 2008), iteratively identifying reliable examples (Ienco & Pensa, 2016), and most notably risk minimization (Northcutt et al., 2017; Du Plessis et al., 2015; Kiryo et al., 2017).
However, all these state-of-the-art approaches share the often-unrealistic assumption that no bias exists in the labeling process, sequential or otherwise.
When a bias is present, these methods are susceptible to learning skewed decision boundaries and thus are prone to making biased and incorrect classiﬁcations
A few recent PU methods have begun to model bias in the labeling process, though none have captured sequential bias.
One approach assumes the likelihood that a given true positive instance is labeled depends solely on its distance from the negative distribution (Kato et al., 2019), while others explore the case when this labeling likelihood depends on the general position of the positive instance in the feature space (Bekker & Davis, 2018a).
Our DeepSPU method extends beyond these methods by coping with both this feature-level bias and the previously-overlooked sequential bias.
P ROBLEM S ETTING 3. P OSITIVE U NLABELED L EARNING
Positive Unlabeled (PU) learning is the task of training a classiﬁer to predict the true class of each instance given a set of mostly unlabeled data and only some labeled positive instances (Bekker &
Davis, 2020).
Solving this problem requires modeling the likelihood that any given true positive instance is labeled.
This labeling likelihood is referred to as the propensity score (Bekker & Davis, 2018a).
Estimating the propensity score is not the same as predicting whether an instance is labeled, which is often relatively easy.
This is because estimating the propensity score corresponds to determining the likelihood that an instance is labeled conditioned on the instance being a true positive .
We cannot directly estimate this value from the data as we do not know whether or not any given unlabeled instance is a true positive .
Formally, let D = { ( X ( j ) , L ( j ) ) } Nj =1 be a dataset D of N sequence pairs, where X ( j ) is a sequence of real values and L ( j ) is a sequence of binary label indicators, | X ( j ) | = |L
( j ) | .
For readability, we drop the superscript j and describe our approach in terms of one sequence.
Let X =
( x 1 , x 2 , ..., x T ) be a sequence of T real values (which we refer to as a sequence of instances ), and L = ( (cid:96) 1 , (cid:96) 2 , ..., (cid:96) T ) be an associated sequence of label indicators such that (cid:96) i
= 1 if x i is labeled positive, and isotherwise.
Additionally, for each feature-label sequence pair there is an unobserved binary true class sequence, Y =
( y 1 , y 2 , ..., y T ) , y i ∈ { 0 , 1 } , representing the underlying classes of the instance.
In addition to y i being unavailable during training, Pr ( y i = 1 | (cid:96) i = 1) = 1 as we assume there are no labels for negative instances, while Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
as we assume not all positive instances are labeled.
We consider both feature-level and sequential biases in the labeling process.
Feature-level bias assumes the propensity score for a positive instance depends on local features of the instance and thus Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
Pr ( (cid:96) i = 1 | y i = 1 , x i ) .
Sequential bias assumes the propensity score of a positive instance can also depend on the label status of preceding instances, (cid:96) 1: i − 1 , and thus
Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
Pr ( (cid:96) i = 1 | y i = 1 , x 1: i , (cid:96) 1: i − 1 ) .
We call a propensity score that captures sequential and feature-level bias a sequential propensity score q i =
Pr ( (cid:96) i = 1 | y i = 1 , x 1: i , (cid:96) 1: i − 1 ) .
Our goal is to train a classiﬁer g θ ( · ) with parameters θ to solve the binary classiﬁcation problem, such that g θ ( X ) =
Pr Y
( Y | X ) .
During training, only the features X and label status indicator L are observed while the true class Y is unobserved.
Notation used summarized in Appendix A.6.
3. B ACKGROUND ON E MPIRICAL PU R ISK M INIMIZATION
In standard positive-negative binary classiﬁcation, the risk of a classiﬁer g θ is given as:
[Equation]
where E p and E p are the distribution over the positive and negative instances respectively, and C + is the loss incurred from predicting g θ ( x ) given that the true instance is positive while C − is the loss incurred from predicting g θ ( x ) when the true instance is negative.
Several recent works have focused on reformulating the above risk into a “positive-unlabeled” risk that takes expectations over the labelled and unlabeled distributions rather than the positive and negative distributions, as the latter two distributions can not be estimated directly from PU data (Kiryo et al., 2017; Du Plessis et al., 2015).
Directly minimizing the empirical PU risk has been successful in the unbiased SCAR setting (Du Plessis et al., 2015; Kiryo et al., 2017)
Additionally, Bekker et al . proposed a PU risk for the SAR setting, where there is bias in the labeling that is a function of the feature values of each instance (Bekker & Davis, 2018a).
For classiﬁer g θ this risk is given as:
[Equation]
where e ( x ) is the propensity score and represents the probability that a positive instance is labeled.
In the formulation proposed by Bekker et al .
the propensity score is only a function of the local feature values x .
A classiﬁer that directly minimizes Bekker’s PU risk has not been proposed for the case where both the propensity core e ( x ) and the posterior P Y ( Y | X ) are unknown.
This is due to the difﬁculty of estimating both the propensity score and the posterior jointly, which arises from the fact that in the above risk the perceived performance of the estimated posterior g θ (and thus the gradients incurred) is based on the estimate of the propensity score and vice versa.
Ergo, a poor estimate of the propensity score can lead the classiﬁer g θ to a poor solution while a poor classiﬁer g θ leads to an inaccurate estimator of the propensity score.
As described in the following section our proposed DeepSPU method does succeed in learning the propensity score and g θ jointly by minimizing the PU risk directly.
DeepSPU overcomes the aforementioned difﬁculty of training the two latent variables through risk minimization by using a novel iterative training algorithm coupled with additional regularization terms.
M ETHODOLOGY 4. O VERVIEW
We now describe a general estimation procedure for sequentially biased PU data, along with a speciﬁc model for learning in this setting.
4. S EQUENTIAL B IAS L EARNING S TRATEGY
There are 2 major components to our proposed learning strategy.
First, we employ an innovative
Iterative Learning Strategy, iteratively training a classiﬁer model and a propensity score model by minimizing the positive unlabeled risk (Bekker & Davis, 2020).
This is achieved without explicit feedback of the two estimated latent variables.
Second, we design two novel PU cost (regularization) terms that are employed during the iterative training.
These cost terms prevent the networks from converging on naive solutions which minimize the PU risk but do not result in correct probability distributions for the latent target variables.
4.2. I TERATIVE L EARNING S TRATEGY
As stated in section 3.2 typical classiﬁers for fully-labeled data are trained to minimize the expected value of a loss function C , known as the risk .
If the propensity score is known then the standard risk
R can be expressed in terms of expectations over only positive and unlabeled distributions, instead of the positive and negative distributions (Du Plessis et al., 2015; Bekker & Davis, 2018a).
Thus, to train our model we express the empirical positive unlabeled risk , R PU , in terms of our novel sequential propensity score q i as:
[Equation]
where C + is the loss incurred from predicting g ( x ) assuming that the true class is positive, C − is the loss incurred from predicting g ( x ) assuming the true class is negative, and q i
=
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 ) is the propensity score of the i -th instance.
If our propensity scores are accurate, then minimizing the above equation will correspond to minimizing the true risk.
This means that in effect we can learn the same classiﬁer that we would have found if we had been given fully labeled data.
Theorem 1.
If q i =
Pr ( (cid:96) = 1 | y i = 1 , x 1: i , (cid:96) 1: i − 1 ) for i = 1 : T , then R pu ( g, q | X, L ) is an unbiased estimation of the true risk R ( g | Y ) .
A proof of this theorem is given in Section A.4 of the Appendix.
As stated, Theorem 1 indicates that we can train a classiﬁer on positive and unlabeled data if we have accurate propensity scores.
However, the propensity score is in general an unobserved observed variable to use as a target during training, and thus there is no straightforward way to minimize the PU risk.
The naive approach would be to simply learn the parameters of the classiﬁer g and propensity model q simultaneously by minimizing Equation 2.
However, there is no guarantee that Equation 2 is an unbiased estimate of the true risk if the estimated propensity scores are incorrect (and thus no indication that this would yield a good classiﬁcation model).
We propose to overcome this challenge with a novel Iterative
Learning Strategy , outlined below.
We begin training the a classiﬁcation model and propensity score model by initializing the propensity scores with “good” estimates, acquired by assuming the labeling likelihood c
=
Pr ( (cid:96) i = 1 | y i = 1) is constant for all positive instances.
c is easily computed given a prior on the class π =
Pr ( y = 1) (Bekker & Davis, 2020), and has been shown to be successfully estimated from the initial data set (Jain et al., 2016).
We show how to derive c from π in the appendix.
Next, we train a propensity model and a classiﬁcation model iteratively in separate interleaved stages.
That is, at each stage of training, either the parameters of the propensity model or classiﬁcation model are updated independently from each other.
4.2. R EGULARIZING C OST T ERMS
It is possible to minimize Equation 2 by naively setting q
i
=
1 ∀ i and g ( x i ) =
(cid:96) i .
In this case, the classiﬁer does not predict the instance’s true class , and instead erroneously predicts whether or not an instance is labeled (Bekker & Davis, 2018a).
To avoid this adversarial solution, we propose two cost terms.
First, we add the Prior-Matching Costs (PMC) , which drives the percentage of predicted positives to match the percentage of true positive we expect according to our prior.
The
PMC is given by the KL divergence between the unconditioned distribution of predicted positives and Bern ( π ) , the Bernoulli distribution parameterized by the true class prior π :
[Equation]
Thus, the number of predicted positives are driven to match the expected number of true positives.
By deﬁnition, the PMC will be larger than the number of labeled instances, and thus Equation Twill incur a high penalty cost for the adversarial solution .
Additionally, if the estimated propensity scores match the true propensity scores, then our propensity score multiplied by the class probability of the i -th instance will equal the probability that the i -th instance is labeled, as stated in Theorem 2.
This informs our next cost term.
Theorem 2.
If q i =
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 , y i = 1) is the sequential propensity score of the i -th instance, then Pr ( (cid:96) i =
| x i , (cid:96) i − 1 )
=
q i · Pr ( y i | x i ) .
The proof is given in Appendix A.5.
In short, we propose to explicitly encourage the product of the estimated propensity scores and predicted class probabilities to match the observed labels.
We accomplish this by adding the Binary
Cross Entropy (BCE) between them as next cost term.
We refer to this as the Observation-Matching
Costs (OMC) , as it requires our propensity score and class predictions to match the observed data:
[Equation]
We thus propose a ﬁnal combined cost function sequentially biased PU data as follows:
[Equation]
where I is an indicator variable that equals 1 if we’re updating the parameters of the classiﬁcation model g Θ and is 0 otherwise, and J is the cost term deﬁned as:
[Equation]
with λ 1 and λ 2 being weights on the corresponding cost terms and J a ( b ) corresponding to the above equation as a function of b while a is held constant.
4. T HE D EEP SPU M ODEL FOR S EQUENTIALLY B IASED D ATA
We now propose a speciﬁc model to minimize Equation 5.
Our model, DeepSPU , consists of three sub-networks: 1) the Representation Network , which learns a robust representation of the input data, 2) the Sequential Propensity Network , which models the likelihood that an instance is labeled given that it is positive, and 3) the Classiﬁcation Network , which models the likelihood that a given instance belongs to the positive class.
The Sequential Propensity Network, used only during the training stage, is crucial in the training of the Classiﬁcation Network as it allows us to train the
Classiﬁcation Network given only weakly-labeled training data.
After training, the Classiﬁcation
Network can be deployed without the Sequential Propensity Network to predict the true class of new instances.
Detailed pseudo-code is available in the appendix.
4.3. R EPRESENTATION N ETWORK
The Sequential Propensity Network and the Classiﬁcation Network both share a common base representation of the input data.
This shared representation is modeled by a recurrent neural network (RNN) B Ω with parameters Ω , such that B Ω takes in the sequence of input data
X = ( x 1 , x 2 , · · · , x T ) and maps each element of the sequence to a corresponding latent representation H =
( h 1 , h 2 , · · · , h T ) .
Each latent representation h i is given by h
i =
F h ( x i , h
i − 1 ) , with the speciﬁc form of F h determined by the choice of RNN network.
Our implementation of DeepSPU uses a Gated Recurrent Unit (GRU) for the RNN (Cho et al., 2014), though the same principles apply to other RNN architectures.
4. S EQUENTIAL P ROPENSITY N ETWORK
The aptly-named Sequential Propensity Network models the propensity score conditioned on the local feature values of each instance and the feature values of all preceding instances, along with the label indicators of all preceding instances.
The ﬁrst component of this sub-network is another GRU that for each label (cid:96) i produces a latent representation
s i conditioned on all previous labels.
In effect, s i summarizes the label subsequence (cid:96) 1 to (cid:96) i into one latent variable.
This GRU is coupled with a feed-forward network (FFN) that takes in the latent label representation s i − 1 , the previous label indicator (cid:96) i − 1 , and the representation of the input features learned by the Representation Network h i in order to produce the propensity score q i of the ith instance.
With Φ representing the learnable parameters of the Sequential Propensity Network Q Φ , Q Φ models the sequential propensity score as Q Φ ( h i , s i − 1 , (cid:96) i − 1 ) =
q i =
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 ) .
4. C LASSIFICATION N ETWORK
The shared hidden representation h
i is also passed as input into the DeepSPU classiﬁer subnetwork , which models the positive class probability given the features of the input data.
Our classiﬁer model g Θ ( x i ) =
P ( y i | x 1: i , Θ) with parameters Θ is given by: g Θ ( x ) =
F g ( B Ω ( x i ); Θ) , where F g is a fully connected network with parameters Θ .
The predicted class value for an instance x i is given by round ( g ( x i )) , where round ( z ) =
if z > 0 .
5 and is 0 otherwise.
We chose 0.5 as the cutoff as g Θ ( x i ) represents the probability that y i = 1 .
E XPERIMENTS 5. D ATA AND E XPERIMENTAL D ETAILS
Datasets.
We evaluate our models and a relevant set of baselines on several real-world sequential human activity recognition datasets: UCI HAR 1 (Anguita et al., 2013), Older Healthy (OH)
HAR 2 (Torres et al., 2013), and ExtraSensory (ES) 3 (Vaizman et al., 2017).
ExtraSensory was subsampled at one reading for every 10 minutes of collected data due to the size of the dataset.
Due to extreme class imbalance, we combine the multiple classes into one for some of the datasets.
[Figure]
We combine UCI HAR into 3 classes: one representing Walking-related activities, one that combines Going Up and Down Stairs, and one that combines stationary activities.
We likewise combine stationary activities in OH HAR .
We also evaluate on RealityCommons (RC Flu) 4 (Madan et al., 2011), an in-the-wild health dataset, where the task is to classify whether the individual is experiencing ﬂu symptoms on a given day using mobile sensor data.
Compared methods.
We compare DeepSPU to the following state-of-the-art methods:
• Positive-Negative (PN) Classiﬁer (Cho et al., 2014):
As a baseline, we adopt a standard binary positive-negative classiﬁer that treats all unlabeled instances as negatives.
The model otherwise has the same structure as the DeepSPU classiﬁcation network.
• uPU (Du Plessis et al., 2015): uPU is a recent highly-inﬂuential approach to training deep networks on PU data.
uPU’s convex approach minimizes the empirical PU risk.
However, this approach assumes the propensity score is constant .
• nnPU (Kiryo et al., 2017): Similarly to uPU , nnPU also assumes that the propensity score is constant to minimize the empirical PU risk.
Additionally, nnPU clips the risk of the unlabeled instances at 0 to avoid inﬁnitely negative risks and is thus liable to overﬁt during training.
• SAR-EM (Bekker & Davis, 2018a): SAR-EM is the leading PU method for learning under feature-level biased labeling .
SAR-EM jointly trains a classiﬁer and propensity network using an Expectation-Maximization algorithm, such that the propensity network can handle feature-level but not sequential bias.
We do not compare against any general semi-supervised methods.
This is not an oversight .
As per our problem deﬁnition, we assume we have no labeled negative examples.
This means general semi-supervised methods are inapplicable.
(Van Engelen & Hoos, 2020).
Evaluation metric.
To measure the performance of our compared methods, we use balanced accuracy (BA) (Brodersen et al., 2010), which is deﬁned as 12 ∗ (cid:16)
[Equation]
BA is similar to the standard accuracy metric, but accounts for class imbalance: a BA of 0.5 is achieved by a naive classiﬁer, regardless of class imbalance.
An optimal classiﬁer will achieve a BA of 1.0.
Implementation details.
We use an 70%/10%/20% train/validation/test split for each dataset.
The base classiﬁer for all methods is a GRU with a 10-dimensional hidden layer.
A 1-layer feed-forward neural network uses the GRU’s latent representation as input for each instance and serves as the classiﬁer.
DeepSPU has an additional 1-layer GRU for the propensity network.
Each method is trained until convergence (200 epochs).
All methods are implemented in PyTorch and are publicly available 5 .
[Table] [Table]
5. E XPERIMENTAL S TUDY ON C LASSIFYING N ATURALLY -U NLABELED D ATA
First, we demonstrate DeepSPU ’s ability to classify data that naturally exhibits sequential bias.
This can be clearly shown for the ExtraSensory and RealityCommons datasets, because both were collected “in the wild” with study participants labeling their own data sequentially.
Thus they are thus naturally weakly labeled.
Further, due to the study design it is unclear if an instance that is not labeled positive is in fact negative or an unlabeled positive (Chang et al., 2017; Madan et al., 2011), thus ﬁtting the positive unlabeled problem description.
For this experiment, we estimate the class prior for nnPU and DeepSPU using TIcE (Bekker & Davis, 2018b).
As shown in Table 1, DeepSPU signiﬁcantly outperforms the state-of-the-art methods for both datasets.
This demonstrates that tackling sequential bias inherent in these real-world datasets is impactful.
Additionally, the baseline underperforms the PU methods in almost all cases, highlighting the importance of leveraging
PU classiﬁers for these weakly-labeled real-world problems.
Overall, these results conﬁrm that the pervasive problem of sequential bias in labeling can be mitigated by DeepSPU .
5. E XPERIMENTAL S TUDY ON L EARNING FROM V ARIOUS L EVELS OF L IMITED L ABELS
In practice, the percentage of labeled positives available during training will vary from dataset to dataset.
Thus, we perform this next experiment to study the impact of the proportion of labeling on each method’s performance.
In line with much of the recent PU work (Bekker & Davis, 2020; 2018a; Kiryo et al., 2017), we achieve this by removing subsets of labels from each dataset prior to training.
We range label availability from 5% to 15% of all positive instances, and remove the labels from all negative instances.
To encourage sequential bias, unlabeling is done sequentially:
A binary Markov Chain decides whether or not to remove the label for each instance in turn.
The likelihood the Markov Chain switches from “labeling” to “unlabeling” states is varied according to the desired level of unlabeling.
Details of this unlabeling process are in Appendix A.7.
As shown in Table 2, DeepSPU signiﬁcantly outperforms all other methods for every level of unlabeling.
This demonstrates that even with very few labels, modeling sequential bias leads to signiﬁcant improvements in performance.
As expected, the Positive Negative performs the worst, as it does not account for unlabeled positives.
Surprisingly, SAR-EM is outperformed by nnPU , despite nnPU assuming that no bias arises in the labeling process.
This may be explained by nnPU ’s natural aversion to overﬁtting (Kiryo et al., 2017), while SAR-EM may learn spurious relationships in the labels while mistakenly modeling sequential bias as feature-level bias.
[Figure]
5. A BLATION STUDY
DeepSPU minimizes PU risk using three components: The iterative training algorithm, the observation-matching cost (OMC), and the prior-matching cost (PMC).
In this experiment, we demonstrate the necessity of each component in an ablation study on the ExtraSensory
Sitting dataset.
Removing any of the three components results in signiﬁcantly lower classiﬁcation accuracy.
Speciﬁcally, removing the iterative training strategy impacts the performance most signiﬁcantly.
This is expected, as without the iterative training strategy the cost function is likely to be biased.
We also see that the PMC divergence, which encourages the percentage of predicted positives to match the class prior, is the more important of the two novel cost terms.
5. E XPERIMENTAL S TUDY E VALUATING P ARAMETERS FOR E STIMATED P RIOR
PU methods rely on prior estimates of class label likelihoods
π = Pr ( y = 1) .
In particular,
DeepSPU uses this prior for the PMC term.
With this value being estimated (Jain et al., 2016), the selected prior could be inaccurate in a practical setting.
Therefore, we now study the impact of poorly-estimated priors on DeepSPU by varying this estimation.
We train DeepSPU on the ES
Sitting dataset, where the true class prior is about 0.36.
As shown in Figure 2b, DeepSPU achieves the strongest performance when the estimated prior is closest to the true prior.
However, even when the prior is overestimated, the performance is not signiﬁcantly impacted until the estimated prior becomes severely over or underestimated.
This indicates that in practice DeepSPU is generally robust even to incorrectly estimated priors.
C ONCLUSION
We propose DeepSPU , the ﬁrst PU solution learning from weakly labeled data with sequential bias in the labeling.
We formulate a novel iterative learning strategy to jointly train a classiﬁcation model and labeling likelihood (propensity) model, along with designing two theoretically-justiﬁed PU cost terms to account for this bias.
Through a series of extensive experimental results we demonstrate that the previously-overlooked sequential labeling bias naturally arises in real-world datasets.
Also, the state-of-the-art PU methods have poor performance when this type of bias is present, while
DeepSPU achieves robust classiﬁcation performance under sequential labeling bias for a rich variety of real-world data sets.
A A PPENDIX A. S TRATEGIES FOR C ALCULATING THE C LASS P RIOR
Our DeepSPU method and the state-of-the-art nnPU method both take in the class prior π = p ( y = 1) as a parameter.
While this can be approximated from PU data (Jain et al., 2016), we calculated this from the true class labels for all experiments.
This was done as follows:
ˆ π = (cid:80) Nn =1 (cid:80) Tt =1 y ( j )
i
NT , where N is the number of time series in the training set and T is the length of each series.
[Equation]
input : Initialized propensity network Q
Initialized classiﬁcation network G
Initialized shared representation B
Dataset of sequences D with features X and labels L of split into batches
Prior estimate ˆ π output: Trained classiﬁer
Trained propensity function c
= num labeled instances ˆ π deﬁne frozen propensity ( x ) =
c train classifier
=
True for i ← 1 to max epochs do for batch in num batches do if train classifier then ˆ Y batch =
G ( B ( X batch ))
ˆ Q batch = frozen propensity ( frozen B ( X batch ))
else ˆ Y batch = frozen classifier ( frozen B ( X batch ))
ˆ Q batch =
Q ( B ( X batch )) end loss ( ˆ Q batch , ˆ Y batch )
=
ˆ R pu + BCE + KL div end
Update weights of B with Adam optimizer using loss if train classifier then
Update weights of G with Adam optimizer using loss else
Update weights of Q with Adam optimizer using loss end train classifier = not train classifier frozen propensity = copy ( Q ) frozen classifier =
copy ( G ) frozen B = copy ( B ) end
For the sake of reproducibility we provide the pseudocode for training DeepSPU.
A. F INDING L ABELING L IKELIHOOD c G IVEN THE C LASS P RIOR
As described in the Methodology section, DeepSPU initially considers the propensity score to be constant for the ﬁrst few epochs of training.
This constant propensity score is found given knowledge of the class prior π as follows:
[Equation]
Thus, we need only to estimate Pr ( (cid:96) = 1) by ﬁnding the fraction of labeled instances in the dataset, and dividing this value by the class prior π .
A. P ROOF OF T HEOREM
Proof.
[Equation]
A. P ROOF OF T HEOREM
Proof.
Pr
[Equation]
(cid:96)
[Equation]
i
[Equation] [Equation]
( (cid:96)
i
[Equation]
1 | x i
[Equation]
(cid:96) 1: i − 1 , y i = 1)
Pr ( y i =
| x i , (cid:96) 1: i − 1 )
+ Pr ( (cid:96) i =
| x i , (cid:96) i − 1 , y i = 0)
Pr ( y i =
| x i , (cid:96) 1: i − 1 ) =
Pr ( (cid:96) i = 1 | x i , (cid:96)
i
[Equation]
1 , y i = 1) Pr ( y i = 1 | x i , (cid:96) 1: i − 1 ) =
Pr ( (cid:96) i = 1 | x i , (cid:96) i − 1 , y i = 1)
Pr ( y i =
| x i )
A.
T ABLE OF N OTATION
[Table]
where i = 1 , · · · , N and t = 1 , · · · , T
A. I NTRODUCING S EQUENTIAL B IAS
For each sub-sequence of consecutive positive instances within each sequence, a binary Markov chain was run to decide which instances were to be unlabeled.
We initiated the Markov chain at state ‘1’, where state ‘1’ indicates that the corresponding positive instance remains labeled.
The
Markov chain could transition to state ‘0’ with probability λ .
We varied the value of λ to evaluate
DeepSPU’s performance for various levels of mislabeling.
For instances where the Markov chain is in state ‘0’ the corresponding positive instance was set to be unlabeled.
The chain has a 1% probability of transitioning back to ‘1’ from state 0.
An example of this is shown in ﬁgure 3.
This process introduces sequential bias, as the likelihood that a given instance is mislabeled depends on whether the previous instance is mislabeled (a given instance has a 100 X (1 - λ ) % chance of being labeled if the preceding instance is labeled).
A. Q UANTITATIVE A NALYSIS ON E XTRA S ENSORY LABELING P ATTERNS
Figure 4 shows the frequency of lengths of labeled subsequences in the ExtraSensory dataset.
Clearly, the participants do enter phases of labeling as there are many long subsequences of consecutive labels.
A. E VALUATING P ERFORMANCE OF P ROPENSITY M ODEL
We perform an additional experiment in order to demonstrate DeepSPU’s propensity network’s ability to learn the true propensity score.
In this experiment, we created subsequences of positive instances and subsequences of negative instances.
The feature values for the positive instances where drawn from a normal distribution with mean 0 and unit variance, while the features for the negative instanced were drawn from a normal distribution with mean 10 and unit variance.
The positive subsequences were labeled using a gamma distribution to decide which positive instances received labels.
We used Scipy’s
(Virtanen et al., 2020) gamma distribution with shape, location, and scale parameters all set to 1.
We then train DeepSPU on this data and extracted its learned propensity scores.
As Figure 5 shows, the estimated propensity scores match the true propensity scores incurred by the gamma distribution almost perfectly.