D EEP P OSITIVE U NLABELED L EARNING WITH A S EQUENTIAL B IAS
Anonymous authors
Paper under double-blind review
A BSTRACT
For many domains, from video stream analytics to human activity recognition, only weakly-labeled datasets are available.
Worse yet, the given labels are often assigned sequentially , resulting in sequential bias .
Current Positive Unlabeled (PU) classiﬁers, a state-of-the-art family of robust semi-supervised methods, are ineffective under sequential bias.
In this work, we propose DeepSPU , the ﬁrst method to address this sequential bias problem.
DeepSPU tackles the two interdependent subproblems of learning both the latent labeling process and the true class likelihoods within one architecture.
We achieve this by developing a novel iterative learning strategy aided by theoretically-justiﬁed cost terms to avoid collapsing into a naive classiﬁer.
Our experimental studies demonstrate that DeepSPU outperforms state-of-the-art methods by over 10% on diverse real-world datasets.
I NTRODUCTION
Background.
Traditional classiﬁers assume that every instance in the training set has been assigned a label.
However, for many real-world problems it is prohibitively expensive to obtain a completely labeled dataset.
Further, there is often bias in the labeling process, which makes some instances more likely to be labeled than others.
For many domains, such as digital healthcare, assisted living smart systems, and video scene description (Aafaq et al., 2019; Ranasinghe et al., 2016), instances are labeled sequentially .
Hence, an instance is more likely to be labeled if its neighbors are also labeled, and vice versa.
Ours is the ﬁrst work to study this setting, which we call sequential bias .
A prevalent example of how sequential bias arises comes from the Human Activity Recognition (HAR) domain (Vaizman et al., 2017).
HAR data are collected by asking subjects to report the activities they perform while wearing mobile sensors.
When collecting long sequences of such data (perhaps many days worth), most participants do not label every activity, and thus much of the data goes unlabeled.
Additionally, since wearable sensors record data rapidly (often many readings per second), large blocks of time get labeled consecutively, resulting in sequential bias.
Many other domains, such as video analysis for intrusion detection or medical records for predicting illnesses, have similar labeling mechanics such that the labels are applied sequentially and are thus also susceptible to sequential labeling bias (Rodr´ıguez-Moreno et al., 2019; Schaekermann et al., 2018).
This is a crucial issue to address, as existing methods exhibit drastically reduced accuracy when sequential bias is not accounted for (as demonstrated in our Experimental Results).
State-of-the-Art.
Positive Unlabeled (PU) classiﬁers are a family of semi-supervised methods that learn from incompletely-labeled data while not requiring any labeled negative examples (Bekker & Davis, 2020; Elkan & Noto, 2008; Li & Liu, 2005;
Hsieh et al., 2015; Du Plessis et al., 2015;
Kiryo et al., 2017; Bekker & Davis, 2018a; Kato et al., 2019).
This is a key strength of PU methods because representative negative examples, typically required by semi-supervised methods, are often not feasible to acquire.
For instance, in the HAR example, there are inﬁnitely many activities that an individual is not performing at any given time.
Consequentially, participants are only expected to provide some positive labels for their activities (Vaizman et al., 2017).
Unfortunately, existing PU methods make unrealistically restrictive simplifying assumptions on how the labels were applied.
Speciﬁcally, they either assume that there is no bias in the labeling process (the probability of a sample being an unlabeled positive instance is uniform) (Elkan & Noto, 2008;
Du Plessis et al., 2015; Kiryo et al., 2017) or otherwise only depends on the local attributes of each instance (Bekker & Davis, 2018a; Kato et al., 2019).
This means that existing methods do not model sequential bias .
And, as we demonstrate in our experiments, these methods are signiﬁcantly negatively impacted when a sequential bias is present.
Problem Description and Technical Challenges.
Given a dataset of sequences, our goal is to predict the true class likelihoods of each instance within the given sequence given a subset of labeled positive instances during training.
In particular, we focus on the difﬁcult case where labels have been assigned with a sequential bias , with sequential bias deﬁned as the case where the likelihood that a positive instance is labeled varies depending on whether its neighboring instances were labeled.
This problem is challenging due to two difﬁcult interdependent subproblems.
First, we have the dependency problem: if we had a model of the latent labeling process (which we call the propensity model ) that allowed us to identify the true unlabeled positive instances, then we could use this propensity model to train a classiﬁer to produce the true class likelihoods.
However, we need these same true class likelihoods in order to train the propensity model - causing a cyclic dependency.
Second, standard maximum likelihood estimation inherently assumes all instances are labeled, leading to a naive classiﬁer in the presence of labeling bias.
To capture unlabeled positive instances, a PU classiﬁer must instead predict an appropriate number of positive instances without simply assuming all positives instances are labeled.
Our Approach: DeepSPU.
We propose Deep Sequential PU ( DeepSPU ), which is the ﬁrst Positive Unlabeled method to use a propensity score model that predicts the likelihood that any given positive instance is labeled while taking sequential bias into account.
The propensity score allows us to train a classiﬁer network given only partially labeled data.
We achieve this by developing a novel learning method that overcomes the cyclic dependency problem by iteratively learning the propensity score model and the classiﬁer using weakly-labeled data.
Further, we introduce the two novel PU cost terms: the Prior-Matching Costs (PWC) and the Observation-Matching Costs (OMC), which prohibit the propensity model and classiﬁer from collapsing into incorrect naive solutions.
Contributions.
The main contributions of our work are:
• We identify sequential bias , a labeling pattern characteristic of many real-world labeling processes, and demonstrate how ignoring this bias signiﬁcantly impacts the performance of state-ofthe-art PU classiﬁers.
• We develop DeepSPU, a novel iterative learning method that tackles sequential bias by jointly learning the two interdependent latent variables: the propensity score and the true class probabilities - even without any direct supervision for either learning task.
• We design the Prior-Weghted Costs and the Observation-Matching Costs, two novel PU cost terms justiﬁed by theoretical analysis that prohibit the classiﬁer and propensity score model to collapsing to incorrect adversarial solutions.
R ELATED W ORK
PU classiﬁers, a special type of semi-supervised models, learn from weakly-labeled training data without requiring any labeled negative examples (Bekker & Davis, 2020).
In contrast, semisupervised methods lean heavily on having access to both labeled positive and labeled negatives to learn from weakly supervised data, making them less robust than PU alternatives (Van Engelen & Hoos, 2020).
PU methods are also more robust than a similar family of classiﬁers known as One
Class (OC) classiﬁers, as general OC classiﬁers do not make use of unlabeled data during training (Khan & Madden, 2009).
There are many approaches to PU learning, such as re-weighting predictions (Zhang & Lee, 2005;
Elkan & Noto, 2008), iteratively identifying reliable examples (Ienco & Pensa, 2016), and most notably risk minimization (Northcutt et al., 2017; Du Plessis et al., 2015; Kiryo et al., 2017).
However, all these state-of-the-art approaches share the often-unrealistic assumption that no bias exists in the labeling process, sequential or otherwise.
When a bias is present, these methods are susceptible to learning skewed decision boundaries and thus are prone to making biased and incorrect classiﬁcations (Bekker & Davis, 2020).
A few recent PU methods have begun to model bias in the labeling process, though none have captured sequential bias.
One approach assumes the likelihood that a given true positive instance is labeled depends solely on its distance from the negative distribution (Kato et al., 2019), while others explore the case when this labeling likelihood depends on the general position of the positive instance in the feature space (Bekker & Davis, 2018a).
Our DeepSPU method extends beyond these methods by coping with both this feature-level bias and the previously-overlooked sequential bias.
P ROBLEM S ETTING
Positive Unlabeled (PU) learning is the task of training a classiﬁer to predict the true class of each instance given a set of mostly unlabeled data and only some labeled positive instances (Bekker &
Davis, 2020).
Solving this problem requires modeling the likelihood that any given true positive instance is labeled.
This labeling likelihood is referred to as the propensity score (Bekker & Davis, 2018a).
Estimating the propensity score is not the same as predicting whether an instance is labeled, which is often relatively easy.
This is because estimating the propensity score corresponds to determining the likelihood that an instance is labeled conditioned on the instance being a true positive .
We cannot directly estimate this value from the data as we do not know whether or not any given unlabeled instance is a true positive .
Formally, let D = { ( X ( j ) , L ( j ) ) } Nj =1 be a dataset D of N sequence pairs, where X ( j ) is a sequence of real values and L ( j ) is a sequence of binary label indicators, (cid:107) X
( j ) (cid:107) =
(cid:107)L ( j ) (cid:107) .
For readability, we drop the superscript j and describe our approach in terms of one sequence.
Let X =
( x 1 , x 2 , ..., x T ) be a sequence of T real values (which we refer to as a sequence of instances ), and L = ( (cid:96) 1 , (cid:96) 2 , ..., (cid:96) T ) be an associated sequence of label indicators such that (cid:96) i
= 1 if x i is labeled positive, and isotherwise.
Additionally, for each feature-label sequence pair there is an unobserved binary true class sequence, Y =
( y 1 , y 2 , ..., y T ) , y i ∈ { 0 , 1 } , representing the underlying classes of the instance.
In addition to y i being unavailable during training, Pr ( y i = 1 | (cid:96) i = 1) = 1 as we assume there are no labels for negative instances, while Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
as we assume not all positive instances are labeled.
We consider both feature-level and sequential biases in the labeling process.
Feature-level bias assumes the propensity score for a positive instance depends on local features of the instance and thus Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
Pr ( (cid:96) i = 1 | y i = 1 , x i ) .
Sequential bias assumes the propensity score of a positive instance can also depend on the label status of preceding instances, (cid:96) 1: i − 1 , and thus
Pr ( (cid:96) i = 1 | y i = 1) (cid:54) =
Pr ( (cid:96) i = 1 | y ( j ) i
= 1 , x 1: i , (cid:96) 1: i − 1 ) .
We call a propensity score that captures sequential and feature-level bias a sequential propensity score q i =
Pr ( (cid:96) i = 1 | y i = 1 , x 1: i , (cid:96) 1: i − 1 ) .
Modeling q
i explicitly enables identiﬁcation of likely true positives among the unlabeled data.
More speciﬁcally, an unlabeled instance with a very low propensity score is likely a true positive.
Our goal is to train a classiﬁer g θ ( · ) with parameters θ to solve the binary classiﬁcation problem, such that g θ ( X ) =
Pr ( Y | X ) .
During training, only the features X and label status indicator L are observed while the true class Y is unobserved.
Notation used summarized in Appendix A.6.
M ETHODOLOGY 4. O VERVIEW
Our proposed model, DeepSPU , consists of three sub-networks: 1) the Representation Network , which learns a robust representation of the input data, 2) the Sequential Propensity Network , which models the likelihood that an instance is labeled given that it is positive, and 3) the Classiﬁcation
Network , which models the likelihood that a given instance belongs to the positive class.
The Sequential Propensity Network, used only during the training stage, is crucial in the training of the Classiﬁcation Network as it allows us to train the Classiﬁcation Network given only weaklylabeled training data.
We employ an innovative Iterative Learning Strategy, training the two prediction networks by minimizing the positive unlabeled risk (Bekker & Davis, 2020).
This is achieved without explicit feedback for each of the two networks’ estimated latent variables.
Further, we design two novel PU cost (regularization) terms that are employed during the iterative training.
These cost terms prevent the networks from converging on naive solutions which minimize the PU risk but
[Figure]
do not result in correct probability distributions for the latent target variables.
After training, the
Classiﬁcation Network can be deployed without the Sequential Propensity Network to predict the true class of new instances.
Detailed pseudo-code is available in the appendix.
4. R EPRESENTATION N ETWORK
The Sequential Propensity Network and the Classiﬁcation Network both share a common base representation of the input data.
This shared representation is modeled by a recurrent neural network (RNN) B Ω with parameters Ω , such that B Ω takes in the sequence of input data
X = ( x 1 , x 2 , · · · , x T ) and maps each element of the sequence to a corresponding latent representation H =
( h 1 , h 2 , · · · , h T ) .
Each latent representation h i is given by h
i =
F h ( x i , h
i − 1 ) , with the speciﬁc form of F h determined by the choice of RNN network.
Our implementation of DeepSPU uses a Gated Recurrent Unit (GRU) for the RNN (Cho et al., 2014), though the same principles apply to other RNN architectures.
4. S EQUENTIAL P ROPENSITY N ETWORK
The aptly-named Sequential Propensity Network models the propensity score conditioned on the local feature values of each instance along with label indicator of all preceding instances.
The ﬁrst component of this sub-network is another GRU that for each label (cid:96) i produces a latent representation
s i conditioned on all previous labels.
In effect, s i summarizes the label subsequence (cid:96) 1 to (cid:96) i into one latent variable.
This GRU is coupled with a feed-forward network (FFN) that takes in the latent label representation s i − 1 , the previous label indicator (cid:96) i − 1 , and the representation of the input features learned by the Representation Network h i in order to produce the propensity score q i of the ith instance.
With Φ representing the learnable parameters of the Sequential Propensity Network Q Φ ,
Q Φ models the sequential propensity score as Q Φ ( h i , s i − 1 , (cid:96) i − 1 )
=
q i =
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 ) .
4. C LASSIFICATION N ETWORK
The shared hidden representation h
i is also passed as input into the DeepSPU classiﬁer subnetwork , which models the positive class probability given the features of the input data.
Our classiﬁer model g Θ ( x i ) =
p ( y i | x 1: i , Θ) with parameters Θ is given by: g Θ ( x ) =
F g ( B Ω ( x i ); Θ) , where F g is a fully connected network with parameters Θ .
The predicted class value for an instance x i is given by round ( g ( x i )) , where round ( z ) =
if z > 0 .
5 and is 0 otherwise.
We chose 0.5 as the cutoff as g Θ ( x i ) represents the probability that y i = 1 .
4. I TERATIVE L EARNING S TRATEGY
Typical classiﬁers for fully-labeled data are trained to minimize the expected value of a loss function
C given the class predictions.
The expected loss is referred to as the risk .
If the propensity score is known then the standard risk R can be expressed in terms of expectations over only positive and unlabeled distributions, instead of the positive and negative distributions (Du Plessis et al., 2015;
Bekker & Davis, 2018a).
Thus, to train our model we express the empirical positive unlabeled risk ,
R PU , in terms of our novel sequential propensity score q i as:
[Equation]
where C + is the loss incurred from predicting g ( x ) assuming that the true class is positive, C − is the loss incurred from predicting g ( x ) assuming the true class is negative, and q i
=
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 ) is the propensity score of the i -th instance.
If our propensity scores are accurate, then minimizing the above equation will correspond to minimizing the true risk.
This means that in effect we can learn the same classiﬁer that we would have found if we had been given fully labeled data.
Theorem 1.
If q i =
Pr ( (cid:96) = 1 | y i = 1 , x 1: i , (cid:96) 1: i − 1 ) for i = 1 : T , then R pu ( g, q | X, L ) is an unbiased estimation of the true risk R ( g | Y ) .
A proof of this theorem is given in Section A.4 of the Appendix.
As stated, Theorem 1 indicates that we can train a classiﬁer on positive and unlabeled data if we have accurate propensity scores.
However, our model of the propensity score (given by the Sequential Propensity Network) has no observed variable to use as a target during training, and thus there is no straightforward way to minimize the PU risk.
The naive approach would be to simply learn the parameters of the classiﬁer g and propensity network q simultaneously by minimizing Equation 1.
However, there is no guarantee that Equation 1 is an unbiased estimate of the true risk if the estimated propensity scores are incorrect (and thus no indication that this would yield a good classiﬁcation model).
We overcome this challenge with a novel Iterative Learning Strategy , outlined below.
We begin training the networks by initializing the propensity scores with “good” estimates, acquired by assuming the labeling likelihood c
=
Pr ( (cid:96) i = 1 | y i = 1) is constant for all positive instances.
c is easily computed given a prior on the class π =
Pr ( y = 1) (Bekker & Davis, 2020), and has been shown to be successfully estimated from the initial data set (Jain et al., 2016).
We show how to derive c from π in the appendix.
Next, we train the Sequential Propensity Network and the Classiﬁcation Network iteratively in separate interleaved stages.
That is, at each stage of training, either the parameters of the Sequential
Propensity Network or Classiﬁcation Network are updated independently from each other.
The
Representation Network is trained at both stages because the Sequential Propensity Network and
Classiﬁcation Network both rely on its hidden representation.
To avoid the Sequential Propensity
Network and Classiﬁcation Network learning based on different hidden representations than they were trained on, when a network’s weights are not being updated a frozen copy of the Representation Network is used to generate the hidden representation for that second network.
4. T HE DeepSPU PU C OST T ERMS
It is possible to minimize Equation 1 by naively setting q
i
=
1 ∀ i and g ( x i ) =
(cid:96) i .
In this case, the classiﬁer does not predict the instance’s true class , and instead erroneously predicts whether or not an instance is labeled (Bekker & Davis, 2018a).
To avoid this adversarial solution, we propose two cost terms.
First, we add the Prior-Matching Costs (PMC) , which drives the percentage of predicted positives to match the percentage of true positive we expect according to our prior.
The
PMC is given by the KL divergence between the unconditioned distribution of predicted positives and a Bernoulli distribution parameterized by the true class prior π :
[Equation]
Thus, the number of predicted positives are driven to match the expected number of true positives.
By deﬁnition, the PMC will be larger than the number of labeled instances, and thus Equation Twill incur a high penalty cost for the adversarial solution .
Additionally, if our propensity scores match the true propensity scores, then our propensity score multiplied by the class probability of the i -th instance will equal the probability that the i -th instance is labeled, as stated in Theorem 2.
This informs our next cost term.
Theorem 2.
If q i =
Pr ( (cid:96) i = 1 | x 1: i , (cid:96) 1: i − 1 , y i = 1) is the sequential propensity score of the i -th instance, then Pr ( (cid:96) i =
| x i , (cid:96) i − 1 )
=
q i · Pr ( y i | x i ) .
The proof is given in Appendix A.5.
In short, DeepSPU explicitly encourages the product of the propensity network and predicted class probability to match the observed labels.
We accomplish this by adding the Binary Cross Entropy (BCE) between them as next cost term.
We refer to this as the Observation-Matching Costs (OMC) , as it requires our propensity score and class predictions to match the observed data:
[Equation]
The ﬁnal loss function for DeepSPU is thus deﬁned by:
[Equation]
where I is an indicator variable that equals 1 if we’re updating the weights of the classiﬁer network g Θ and is 0 otherwise, and J is the cost term deﬁned as:
[Equation]
with λ 1 and λ 2 being weights on the corresponding cost terms.
E XPERIMENTS 5. D ATA AND E XPERIMENTAL D ETAILS
Datasets.
We evaluate our models and a relevant set of baselines on several real-world sequential human activity recognition datasets: UCI HAR 1 (Anguita et al., 2013), Older Healthy (OH)
HAR 2 (Torres et al., 2013), and ExtraSensory (ES) 3 (Vaizman et al., 2017).
ExtraSensory was subsampled at one reading for every 10 minutes of collected data due to the size of the dataset.
Due to extreme class imbalance, we combine the multiple classes into one for some of the datasets.
We combine UCI HAR into 3 classes: one representing Walking-related activities, one that combines Going Up and Down Stairs, and one that combines stationary activities.
We likewise combine stationary activities in OH HAR .
We also evaluate on RealityCommons (RC Flu) 4 (Madan et al., 2011), an in-the-wild health dataset, where the task is to classify whether the individual is experiencing ﬂu symptoms on a given day using mobile sensor data.
Compared methods.
We compare DeepSPU to the following state-of-the-art methods:
• Positive-Negative (PN) Classiﬁer (Cho et al., 2014):
As a baseline, we adopt a standard binary positive-negative classiﬁer that treats all unlabeled instances as negatives.
The model otherwise has the same structure as the DeepSPU classiﬁcation network.
• uPU (Du Plessis et al., 2015): uPU is a recent highly-inﬂuential approach to training deep networks on PU data.
uPU’s convex approach minimizes the empirical PU risk.
However, this approach assumes the propensity score is constant .
• nnPU (Kiryo et al., 2017): Similarly to uPU , nnPU also assumes that the propensity score is constant to minimize the empirical PU risk.
Additionally, nnPU clips the risk of the unlabeled instances at 0 to avoid inﬁnitely negative risks and is thus liable to overﬁt during training.
• SAR-EM (Bekker & Davis, 2018a): SAR-EM is the leading PU method for learning under feature-level biased labeling .
SAR-EM jointly trains a classiﬁer and propensity network using an Expectation-Maximization algorithm, such that the propensity network can handle feature-level but not sequential bias.
We do not compare against any general semi-supervised methods.
This is not an oversight .
As per our problem deﬁnition, we assume we have no labeled negative examples.
This means general semi-supervised methods are inapplicable.
(Van Engelen & Hoos, 2020).
[Table]
Evaluation metric.
To measure the performance of our compared methods, we use balanced accuracy (BA) (Brodersen et al., 2010), which is deﬁned as 12 ∗ (cid:16)
[Equation]
BA is similar to the standard accuracy metric, but accounts for class imbalance: a BA of 0.5 is achieved by a naive classiﬁer, regardless of class imbalance.
An optimal classiﬁer will achieve a BA of 1.0.
Implementation details.
We use an 70%/10%/20% train/validation/test split for each dataset.
The base classiﬁer for all methods is a GRU with a 10-dimensional hidden layer.
A 1-layer feed-forward neural network uses the GRU’s latent representation as input for each instance and serves as the classiﬁer.
DeepSPU has an additional 1-layer GRU for the propensity network.
Each method is trained until convergence (200 epochs).
All methods are implemented in PyTorch and are publicly available 5 .
5. E XPERIMENTAL S TUDY ON C LASSIFYING N ATURALLY -U NLABELED D ATA
First, we demonstrate DeepSPU ’s ability to classify data that naturally exhibits sequential bias.
This can be clearly shown for the ExtraSensory and RealityCommons datasets, because both were collected “in the wild” with study participants labeling their own data sequentially.
Thus they are thus naturally weakly labeled.
Further, due to the study design it is unclear if an instance that is not labeled positive is in fact negative or an unlabeled positive (Chang et al., 2017; Madan et al., 2011), thus ﬁtting the positive unlabeled problem description.
For this experiment, we estimate the class prior for nnPU and DeepSPU using TIcE (Bekker & Davis, 2018b).
As shown in Table 1, DeepSPU signiﬁcantly outperforms the state-of-the-art methods for both datasets.
This demonstrates that tackling sequential bias inherent in these real-world datasets is impactful.
Additionally, the baseline underperforms the PU methods in almost all cases, highlighting the importance of leveraging
PU classiﬁers for these weakly-labeled real-world problems.
Overall, these results conﬁrm that the pervasive problem of sequential bias in labeling can be mitigated by DeepSPU .
5. E XPERIMENTAL S TUDY ON L EARNING FROM V ARIOUS L EVELS OF L IMITED L ABELS
In practice, the percentage of labeled positives available during training will vary from dataset to dataset.
Thus, we perform this next experiment to study the impact of the proportion of labeling on each method’s performance.
In line with much of the recent PU work (Bekker & Davis, 2020; 2018a; Kiryo et al., 2017), we achieve this by removing subsets of labels from each dataset prior to training.
We range label availability from 5% to 15% of all positive instances, and remove the labels from all negative instances.
To encourage sequential bias, unlabeling is done sequentially:
A binary Markov Chain decides whether or not to remove the label for each instance in turn.
The likelihood the Markov Chain switches from “labeling” to “unlabeling” states is varied according to the desired level of unlabeling.
Details of this unlabeling process are in Appendix A.7.
As shown in Table 2, DeepSPU signiﬁcantly outperforms all other methods for every level of unlabeling.
This demonstrates that even with very few labels, modeling sequential bias leads to signiﬁcant improvements in performance.
As expected, the Positive Negative performs the worst, as it does not account for unlabeled positives.
Surprisingly, SAR-EM is outperformed by nnPU , despite nnPU assuming that no bias arises in the labeling process.
This may be explained by nnPU ’s natural aversion to overﬁtting (Kiryo et al., 2017), while SAR-EM may learn spurious relationships in the labels while mistakenly modeling sequential bias as feature-level bias.
[Table]
various levels of labeling on the Older Healthy, UCI HAR, and ExtraSensory HAR datasets
Balanced Accuracy (higher is better)
[Figure]
5. A BLATION STUDY
DeepSPU minimizes PU risk using three components: The iterative training algorithm, the observation-matching cost (OMC), and the prior-matching cost (PMC).
In this experiment, we demonstrate the necessity of each component in an ablation study on the ExtraSensory
Sitting dataset.
Removing any of the three components results in signiﬁcantly lower classiﬁcation accuracy.
Speciﬁcally, removing the iterative training strategy impacts the performance most signiﬁcantly.
This is expected, as without the iterative training strategy the cost function is likely to be biased.
We also see that the PMC divergence, which encourages the percentage of predicted positives to match the class prior, is the more important of the two novel cost terms.
5. E XPERIMENTAL S TUDY E VALUATING P ARAMETERS FOR E STIMATED P RIOR
PU methods rely on prior estimates of class label likelihoods
π = Pr ( y = 1) .
In particular,
DeepSPU uses this prior for the PMC term.
With this value being estimated (Jain et al., 2016), the selected prior could be inaccurate in a practical setting.
Therefore, we now study the impact of poorly-estimated priors on DeepSPU by varying this estimation.
We train DeepSPU on the ES
Sitting dataset, where the true class prior is about 0.36.
As shown in Figure 2b, DeepSPU achieves the strongest performance when the estimated prior is closest to the true prior.
However, even when the prior is overestimated, the performance is not signiﬁcantly impacted until the estimated prior becomes severely over or underestimated.
This indicates that in practice DeepSPU is generally robust even to incorrectly estimated priors.
C ONCLUSION
We propose DeepSPU , the ﬁrst PU solution learning from weakly labeled data with sequential bias in the labeling.
We formulate a novel iterative learning strategy to jointly train a classiﬁcation model and labeling likelihood (propensity) model, along with designing two theoretically-justiﬁed PU cost terms to account for this bias.
Through a series of extensive experimental results we demonstrate that the previously-overlooked sequential labeling bias naturally arises in real-world datasets.
Also, the state-of-the-art PU methods have poor performance when this type of bias is present, while
DeepSPU achieves robust classiﬁcation performance under sequential labeling bias for a rich variety of real-world data sets.
A A PPENDIX A. S TRATEGIES FOR C ALCULATING THE C LASS P RIOR
Our DeepSPU method and the state-of-the-art nnPU method both take in the class prior π = p ( y = 1) as a parameter.
While this can be approximated from PU data (Jain et al., 2016), we calculated this from the true class labels for all experiments.
This was done as follows:
ˆ π = (cid:80) Nn =1 (cid:80) Tt =1 y ( j )
i
NT , where N is the number of time series in the training set and T is the length of each series.
[Equation]
input : Initialized propensity network Q
Initialized classiﬁcation network G
Initialized shared representation B
Dataset of sequences D with features X and labels L of split into batches
Prior estimate ˆ π output: Trained classiﬁer
Trained propensity function c
= num labeled instances ˆ π deﬁne frozen propensity ( x ) =
c train classifier
=
True for i ← 1 to max epochs do for batch in num batches do if train classifier then ˆ Y batch =
G ( B ( X batch ))
ˆ Q batch = frozen propensity ( frozen B ( X batch ))
else ˆ Y batch = frozen classifier ( frozen B ( X batch ))
ˆ Q batch =
Q ( B ( X batch )) end loss ( ˆ Q batch , ˆ Y batch )
=
ˆ R pu + BCE + KL div end
Update weights of B with Adam optimizer using loss if train classifier then
Update weights of G with Adam optimizer using loss else
Update weights of Q with Adam optimizer using loss end train classifier = not train classifier frozen propensity = copy ( Q ) frozen classifier =
copy ( G ) frozen B = copy ( B ) end
For the sake of reproducibility we provide the pseudocode for training DeepSPU.
A. F INDING L ABELING L IKELIHOOD c G IVEN THE C LASS P RIOR
As described in the Methodology section, DeepSPU initially considers the propensity score to be constant for the ﬁrst few epochs of training.
This constant propensity score is found given knowledge of the class prior π as follows:
[Equation]
Thus, we need only to estimate Pr ( (cid:96) = 1) by ﬁnding the fraction of labeled instances in the dataset, and dividing this value by the class prior π .
A. P ROOF OF T HEOREM
Proof.
[Equation]
A. P ROOF OF T HEOREM
Proof.
Pr
[Equation]
(cid:96)
[Equation]
i
[Equation] [Equation]
( (cid:96)
i
[Equation]
1 | x i
[Equation]
(cid:96) 1: i − 1 , y i = 1)
Pr ( y i =
| x i , (cid:96) 1: i − 1 )
+ Pr ( (cid:96) i =
| x i , (cid:96) i − 1 , y i = 0)
Pr ( y i =
| x i , (cid:96) 1: i − 1 ) =
Pr ( (cid:96) i = 1 | x i , (cid:96) i − 1 , y i = 1)
Pr ( y i =
| x i , (cid:96) 1: i − 1 ) =
Pr ( (cid:96) i = 1 | x i , (cid:96) i − 1 , y i = 1)
Pr ( y i =
| x i )
A. T ABLE OF N OTATION
[Table]
where i = 1 , · · · , N and t = 1 , · · · , T
A. I NTRODUCING S EQUENTIAL B IAS
For each sub-sequence of consecutive positive instances within each sequence, a binary Markov chain was run to decide which instances were to be unlabeled.
We initiated the Markov chain at state ‘1’, where state ‘1’ indicates that the corresponding positive instance remains labeled.
The
Markov chain could transition to state ‘0’ with probability λ .
We varied the value of λ to evaluate
DeepSPU’s performance for various levels of mislabeling.
For instances where the Markov chain is in state ‘0’ the corresponding positive instance was set to be unlabeled.
The chain has a 1% probability of transitioning back to ‘1’ from state 0.
An example of this is shown in ﬁgure 3.
This process introduces sequential bias, as the likelihood that a given instance is mislabeled depends on whether the previous instance is mislabeled (a given instance has a 100 X (1 - λ ) % chance of being labeled if the preceding instance is labeled).
A. Q UANTITATIVE A NALYSIS ON E XTRA S ENSORY LABELING P ATTERNS
Figure 4 shows the frequency of lengths of labeled subsequences in the ExtraSensory dataset.
Clearly, the participants do enter phases of labeling as there are many long subsequences of consecutive labels.