L EARNING FROM A SYMMETRICALLY - CORRUPTED D ATA IN R EGRESSION FOR S ENSOR M AGNITUDE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper addresses a regression problem in which output label values represent the results of sensing the magnitude of a phenomenon.
A low value of such labels can either mean that the actual magnitude of the phenomenon has been low or that the sensor has made an incomplete observation.
This leads to a bias toward lower values in labels and its resultant learning because labels for incomplete observations are recorded as lower than those for typical observations, even if both have monitored similar phenomena.
Moreover, because an incomplete observation does not provide any tags indicating incompleteness, we cannot eliminate or impute them.
To address this issue, we propose a learning algorithm that explicitly models the incomplete observations to be corrupted with an asymmetric noise that always has a negative value.
We show that our algorithm is unbiased with a regression learned from the uncorrupted data that does not involve incomplete observations.
We demonstrate the advantages of our algorithm through numerical experiments
I NTRODUCTION
This paper addresses a regression problem for predicting the magnitude of a phenomenon when an observed magnitude involves a particular measurement error.
The magnitude typically represents how large a phenomenon is or how strong the nature of the phenomenon is .
Such examples of predicting the magnitude are found in several application areas, including pressure, vibration, and temperature (Vandal et al., 2017; Shi et al., 2017; Wilby et al., 2004; Tanaka et al., 2019).
In medicine and healthcare, the magnitude may represent pulsation, respiration, or body movements (Inan et al., 2009;
Nukaya et al., 2010; Lee et al., 2016; Alaziz et al., 2016; 2017; Carlson et al., 2018).
More speciﬁcally, we learn a regression function to predict the label representing the magnitude of a phenomenon from explanatory variables .
The training data consists of pairs of the label and explanatory variables, but note that the label in the data is observed with a sensor and is not necessarily in agreement with the actual magnitude of the phenomenon.
We note that we use the term “label” even though we address the regression problem, and it refers to a real-valued label in this paper.
In the example of predicting the magnitude of body movements, the label in the data is measured with an intrusive sensor attached to the chest or the wrist, and the explanatory variables are the values measured with non-intrusive bed sensors (Mullaney et al., 1980; Webster et al., 1982; Cole et al., 1992; Tryon, 2013).
A regression function for this example would make it possible to replace intrusive sensors with non-intrusive ones, which in turn will reduce the burden on patients.
Although the sensors that measure the label generally have high accuracy, they often make incomplete observations, and such incomplete observations are recorded as low values instead of missing values .
This leads to the particular challenge where a low value of the label can either mean that the actual magnitude of the phenomenon has been low or that the sensor has made an incomplete observation, and there are no clues that allow us to tell which is the case.
We illustrate this challenge in Fig.
1-(a).
Such incomplete observations are prevalent in measuring the magnitude of a phenomenon.
For example, the phenomenon may be outside the coverage of a sensor, or the sensing system may experience temporal mechanical failures.
In the example of body movements, the sensor may be temporarily detached from the chest or wrist.
In all cases, the sensor keeps recording low values, while the actual magnitude may be high, and no tag indicating incompleteness can be provided.
[Figure]
This incomplete observation is particularly severe for the sensor measuring the label since it is single-source and has narrower data coverage.
This stems from the fact that the sensor is usually intrusive or it is costly to produce highly accurate observations for measuring the label.
Examples of this can be seen in chest or wrist sensors that focus on the movements of a local body part with high accuracy and often miss movements outside their coverage, such as those of parts located far from where the sensor is attached.
At most, a single intrusive sensor can be attached to a patient to avoid burdening them.
In contrast, the sensors measuring the explanatory variables are usually multi-source and provide broader data coverage.
For example, multiple sensors can be attached to various places of a bed and globally monitor the movements of all body parts on the bed but with lower accuracy.
One cannot simply ignore the problem that the observations of labels may be incomplete because the estimated regression functions trained on such data with incomplete observations are severely biased toward lower values regardless of the amount of available training data.
This bias comes from the fact that incomplete observations always have lower values than the actual magnitude of a phenomenon, and they occur intensively on label sensors, while explanatory variables are usually observed completely.
Moreover, incomplete observations can be much more frequent than expected.
Unfortunately, since we cannot identify which observations are incomplete, we cannot eliminate or impute them by using existing methods that require identifying incomplete observations.
Such methods include thresholding, missing value detection (Pearson, 2006; Qahtan et al., 2018), imputation (Enders, 2010; Smieja et al., 2018; Ma & Chen, 2019;
Sportisse et al., 2020), and semi-supervised regression (Zhou & Li, 2005; Zhu & Goldberg, 2009; Jean et al., 2018;
Zhou et al., 2019).
The issues of incomplete observations also cannot be solved with robust regression (Huber et al., 1964; Narula & Wellington, 1982; Draper & Smith, 1998;
Wilcox, 1997), which takes into account the possibility that the observed labels contain outliers.
While robust regression is an established approach and state-of-the-art against corrupted labels in regression, it assumes symmetric label corruption .
Namely, the noise is assumed to not be biased either positively or negatively.
Since incomplete observations induce the noise that is severely biased toward lower values, robust regression methods still produce regression functions that are biased toward lower values than the one that would be learned from the data without incomplete observations.
In this paper, to mitigate the bias toward lower values, we explicitly assume the existence of the noise from incomplete observations, which always has negative values, in addition to the ordinary symmetric noise.
That is, we consider our training data to be asymmetrically-corrupted data .
We then formulate a regression problem from our asymmetrically-corrupted data and design a principled learning algorithm for this regression problem.
By explicitly modeling the incomplete observation, we derive a learning algorithm that has a rather drastic feature: namely, it ignores the labels that have relatively low values (lower-side labeled data).
In other words, our algorithm uses the data whose labels have relatively high values (upper-side labeled data) and the data whose labels are ignored (unlabeled data).
Hence, we refer to our algorithm as upper and unlabeled regression (U2 regression).
This aligns with the intuition that the labels with low values are unreliable, since those low values may be due to incomplete observations.
Our main result is that U2 regression, which learns from the asymmetrically-corrupted data, produces a regression function that is, under some technical assumptions, unbiased and consistent with the one that is produced from the uncorrupted data that does not involve incomplete observations.
This counterintuitive result is achieved by considering a speciﬁc class of loss functions and deriving their gradient , which only requires upper-side labeled data and unlabeled data in the asymmetricallycorrupted data and can still be shown to be asymptotically equivalent to the expression of the gradient that has access to the uncorrupted data.
The main novelty in our approach is thus in the loss function, and we will empirically demonstrate the effectiveness of the proposed class of loss functions over existing common loss functions in dealing with asymmetrically-corrupted data in synthetic and six real-world regression tasks.
Contributions.
The main contributions of this paper are summarized as follows
• We formulate a novel problem of learning a regression function from asymmetricallycorrupted data.
This is important for applications where the magnitude of a phenomenon is measured with a sensor that is susceptible to unidentiﬁable incomplete observations.
• We derive an unbiased and consistent learning algorithm (U2 regression) for this problem from the new class of loss functions.
• Extensive experiments on synthetic and six real-world regression tasks including a real use case for healthcare demonstrate the effectiveness of the proposed method.
R EGRESSION FROM A SYMMETRICALLY - CORRUPTED D ATA
Our goal is to derive a learning algorithm with asymmetrically-corrupted data, i.e., labels in the training data are corrupted with negative-valued noise due to incomplete observations, in a manner that is unbiased and consistent with the regression that uses uncorrupted data without involving incomplete observations.
We ﬁrst consider the regression problem that uses the uncorrupted data in
Section 2.1 and then formulate learning from the asymmetrically-corrupted data in Section 2.2.
2. R EGRESSION P ROBLEM FROM D ATA WITHOUT I NCOMPLETE O BSERVATIONS
Let x ∈ R D ( D ∈ N ) be a D -dimensional explanatory variable and y ∈
R be a real-valued label.
We assume that, without incomplete observations, y is observed in accordance with
[Equation]
where f ∗ is the oracle regressor and (cid:15) s is the symmetric noise with 0 as the center, such as additive white Gaussian noise (AWGN).
We learn a regression function f ( x ) that computes the value of the estimation of a label, ˆ y , for a newly observed x as ˆ y = f
( x ) .
The optimal regression function, ˆ f , is given by
[Equation]
where F is a hypothesis space for f , and L ( f ) is the expected loss when the regression function f ( x ) is applied to data ( x , y ) , distributed in accordance with an underlying distribution p ( x , y ) :
[Equation]
where E p [ • ] denotes the expectation over the distribution p , and L ( f ( x ) , y ) is the loss function between f ( x ) and y , e.g., the squared loss, L ( f ( x ) , y )
= (cid:107) f
( x ) − y (cid:107) 2 .
The expectation E p ( x ,y ) can be estimated by computing a sample average for the training data D ≡ { ( x n , y n ) } Nn =1 , which is N pairs of explanatory variables and labels.
2. R EGRESSION P ROBLEM FROM A SYMMETRICALLY - CORRUPTED D ATA
In this paper, we consider a scenario in which we only have access to the asymmetrically-corrupted data D (cid:48) ≡ { ( x n , y (cid:48) n ) }
Nn =1 , where a label y (cid:48) may be corrupted due to incomplete observations.
A corrupted label y (cid:48) is observed from the uncorrupted y with an asymmetric negative-valued noise, (cid:15) a : y (cid:48) =
y + (cid:15) a , (4) where the asymmetric noise (cid:15) a always has a random negative value, which means y (cid:48) ≤ y .
Using only D (cid:48) , we learn a regression function f ( x ) as the solution for Equation 2 in an unbiased and consistent manner .
Although AWGN can be handled even when we use a naive regression method such as least squares, the asymmetric noise (cid:15) a , which always has a negative value, is problematic.
Intuitively, the asymmetric noise (cid:15) a makes lower-side labeled data particularly unreliable and inappropriate for learning, while keeping upper-side labeled data reliable, where the upper-side labeled data refers to the data { ( x , y ) } whose label is above the regression line (i.e., f ( x ) ≤ y ) and the lower-side labeled data refers to the data whose label is below the regression line.
The regression line represents the estimation of a regression function.
Figure 1-(b) illustrates this as a scatter plot of the value of the label against the value of an explanatory variable.
Here, the data with incomplete observations appear only in the lower side of the regression line because (cid:15) a makes observations have lower label values than those of typical observations, where the regression line represents such typical observations.
This asymmetry leads to biased learning compared with the learning from the uncorrupted data without incomplete observations.
To address the asymmetric noise (cid:15) a and its resultant bias, we formalize the assumption on the observation process for the asymmetrically-corrupted data D (cid:48) and derive a lemma representing the nature of D (cid:48) .
Then, we propose a learning algorithm based on the lemma in the next section.
The observation processes of D and D (cid:48) are formally characterized as follows.
Assumption 2.1.
Assume (cid:15) s ⊥⊥ f ∗ ( x ) , E p ( (cid:15) s )
[ (cid:15) s ] = 0 ; (cid:15) a ⊥⊥ f ∗ ( x ) , (cid:15) a ≤ 0 almost surely (a.s.); 2 | (cid:15) s | < | (cid:15) a | a.s. when (cid:15) a < 0 ; and { ( x n , y, y (cid:48) n ) }
Nn =1 are i.i.d.
observations in accordance with Equation 1 and Equation 4,
This assumption means that D (cid:48) has enough information to estimate f , and the asymmetric noise (cid:15) a is signiﬁcant enough compared to the symmetric noise (cid:15) s , which are necessary assumptions so that the learning problem is solvable, and (cid:15) a should be handled separately from (cid:15) s .
From Assumption 2.1, we then have the following lemma.
Lemma 2.2.
Let F (cid:48) ≡ { f ∈ F : | f ( x ) − f ∗ ( x ) | ≤
| (cid:15) s | a.s. } .
When f ∈ F (cid:48) , the following holds for y ≡ f ∗ ( x ) + (cid:15) s and y (cid:48) ≡ y + (cid:15) a under Assumption 2.1:
[Equation]
(5) for any function G : R D × R → R as long as the expectations exist.
Proof.
We outline a proof here and provide a complete one in Appendix A.1.
We ﬁrst show that (cid:15) a does not change the distribution for upper-side labeled data ( f ∗ ( x ) ≤ y (cid:48) ) on the basis of the oracle regression function f ∗ before and after adding (cid:15) a , i.e., (cid:15) a = 0 when f ∗ ( x ) ≤ y (cid:48) .
With the condition f ∈ F (cid:48) , we can further prove that (cid:15) a = 0 when f ( x ) ≤ y (cid:48) , which is for upper-side labeled data on the basis of f .
This establishes p ( x, y (cid:48) | f ( x ) ≤ y (cid:48) )
= p ( x, y | f ( x ) ≤ y ) and implies Lemma 2.2.
The condition parts of these conditional distributions represent the relationships between labels and the estimations of the regression function f , e.g., p ( x , y | f ( x ) ≤ y ) is the distribution of x and y when y is higher than what is given by f .
The condition f ∈ F (cid:48) represents our natural expectation that the regression function f well approximates f ∗ .
Lemma 2.2 shows that (cid:15) a does not change the expectation for our upper-side labeled data ( f ( x ) ≤ y (cid:48) ) before and after adding (cid:15) a , which makes them still reliable for regression.
In the next section, we derive an unbiased learning algorithm based on this lemma.
U2 R EGRESSION
We seek to ﬁnd the minimizer of the objective in Equation 2 from the asymmetrically-corrupted data D (cid:48) .
To this end, we propose a gradient that relies only on the knowledge of the distribution of the corrupted data p ( x , y (cid:48) ) but is still equivalent to the gradient of Equation 3, which relies on the knowledge of the distribution of the uncorrupted data p ( x , y ) .
Based on Lemma 2.2, we rewrite the gradient based on p ( x , y ) into the one that only requires p ( x , y (cid:48) ) .
3. G RADIENT FOR L EARNING FROM A SYMMETRICALLY - CORRUPTED D ATA
Here, we address Equation 2 with the gradient descent.
At step t + 1 in the gradient descent, the gradient of Equation 3 with respect to the parameters θ of f is represented with a regression function, f t , which is estimated at step t , as follows:
[Equation]
where ∇ L ( f t ( x ) , y ) ≡ ∂L ( f ( x ) , y )
∂ θ (cid:12)(cid:12)(cid:12) f
= f t .
Note that this holds for any step in the gradient descent.
When t = 0 , f 0 is the initial value of f , and when t = ∞ , we suppose f ∞ =
ˆ f .
We can decompose ∇L ( f t ) as
[Equation]
We then assume that, when y < f ( x ) , the gradient of the loss function does not depend on y and only depends on f ( x ) ; thus we write ∇ L ( f ( x ) , y ) as g ( f ( x )) when y < f ( x ) to emphasize this independence.
Formally,
Condition 3.1.
Let g ( f ( x )) be ∇ L ( f ( x ) , y ) for y <
f
( x ) .
g ( f ( x )) is a gradient function depending only on f ( x ) and not on the value of y .
Such common losses are the absolute loss and pinball loss, which are respectively used in least absolute regression and quantile regression and work well on real data (Lee et al., 2016; Yeung et al., 2002; Wang et al., 2005; Srinivas et al., 2020).
For example, the gradient of the absolute loss is
[Equation]
which does not depend on the value of y but only on f ( x )
[Equation]
We now propose a gradient that does not rely on the knowledge of p ( x , y ) but instead uses only p ( x , y (cid:48) ) .
Namely,
[Equation]
In Section 3.2, we will formally establish the equivalence between the gradient in Equation 9 and that in Equation 6 under our assumptions.
Note that in the second and third terms of Equation 9, we apply expectations over p ( x ) and p ( x |
f t ( x ) ≤ y (cid:48) )
to g ( f ( x )) , even though g ( f ( x )) is deﬁned to be the gradient ∇ L ( f ( x ) , y ) for y < f ( x ) .
This is tractable due to the nature of g ( f ( x )) , which only depends on f ( x ) and does not depend on the value of y .
Since the expectations in Equation 9 only depend on x and y (cid:48) , they can be estimated by computing a sample average for our asymmetrically-corrupted data D (cid:48) as
[Equation]
where { X up , y (cid:48) up } represents the set of coupled pairs of x and y (cid:48) in the upper-side labeled sample set, { x, y (cid:48) :
f t ( x ) ≤ y (cid:48) } , in D (cid:48) ; X un is a sample set of x in D (cid:48) ignoring labels y (cid:48) ; n up is the number of samples in the upper-side labeled set; π up is π up ≡ p ( f t ( x ) ≤ y ) .
Note that π up depends on the current estimation of the function f t and the label y with complete observation .
Thus, it changes at each step of the gradient descent, and we cannot determine its value in a general way.
In this paper, we propose a simple approach of choosing π up as a single value of the hyperparameter.
We optimize it with the grid search based on the validation set, which enables us to ﬂexibly handle data variation.
We will show that it works practically in our experiments.
As we will show in Section 3.2, we can use Equation 10 to design an algorithm that gives an unbiased and consistent regression function.
By using the gradient in Equation 10, we can optimize Equationand learn the regression function only with upper-side labeled samples and unlabeled samples from
D (cid:48) independent of lower-side labels.
This addresses the issue that our lower-side labeled data is particularly unreliable and leads to overcoming the bias that stems from this unreliable part of the data.
We refer to our algorithm as upper and unlabeled regression (U2 regression).
See Appendix B for the speciﬁc implementation of the algorithm based on stochastic optimization.
The gradient in Equation 10 can be interpreted in an intuitive manner.
The ﬁrst term in Equationhas the effect of minimizing the upper-side loss.
Recall that the upper-side data are not affected by the asymmetric noise under our assumptions.
Thus, U2 regression seeks to learn the regression function f on the basis of this reliable upper-side data.
Notice that the ﬁrst term becomes zero when all of the data points are below f (i.e., y (cid:48)
≤ f t ( x ) , ∀ ( x, y (cid:48) )
∈ D (cid:48) ), since then { X up , y (cid:48) up (cid:9) becomes empty.
The second term thus has the effect of pushing down f at all of the data points so that some data points are above f .
Meanwhile, the third term partially cancels this effect of the second term for the upper-side data to control the balance between the ﬁrst and the second terms.
3. U NBIASEDNESS AND C ONSISTENCY OF G RADIENT
U2 regression is the learning algorithm based on the gradient, ∇ ˆ L ( f t ) , in Equation 10 and uses only asymmetrically-corrupted data D (cid:48) .
The use of ∇ ˆ L ( f t ) can be justiﬁed as follows:
Proposition 3.2.
Suppose that Assumption 2.1 holds and the loss function L ( f ( x ) , y ) satisﬁes
Condition 3.1.
Then, the gradient ∇ ˜ L ( f t ) in Equation 9 and its empirical approximation ∇ ˆ L ( f t ) in Equation 10 are unbiased and consistent with the gradient ∇L ( f t ) in Equation 6 a.s.
Proof.
We outline a proof here and provide a complete one in Appendix A.2.
First, we rewrite Equation 7 into a gradient that only contains the expectation over p ( x , y | f t ( x ) ≤ y ) with Condition 3.1.
Then, we apply Lemma 2.2 to the gradient, and it becomes an expression identical to Equation 9.
In other words, U2 regression asymptotically produces the same result as the learning algorithm based on the gradient ∇L ( f t ) in Equation 6, which requires the uncorrupted data without incomplete observations, D .
The convergence rate of U2 regression is of the order O p ( 1 / √ n up + 1 / √ N ) in accordance with the central limit theorem (Chung, 1968), where O p denotes the order in probability.
We further justify our approach of having the speciﬁc form of Equation 9 by showing that a straightforward variant that uses D (cid:48) as if it does not involve incomplete observations (i.e., p ( x , y )
≈ p ( x , y (cid:48) ) ) can fail for our problem.
To this end, we introduce an additional assumption on the observation process:
Assumption 3.3.
Assume (cid:15) a ⊥⊥ x .
Then, we have
Lemma 3.4.
Let ∇ ˇ L ( f t ) be a variant of the gradient in Equation 7 replacing p ( x , y ) with p ( x , y (cid:48) ) , δ be the difference between the expectations of the gradients in the upper side and the lower side δ ≡ (cid:12)(cid:12) E p ( x ,y | f
( x ) ≤ y )
[ ∇ L ( f ( x ) , y )] − E p ( x ,y | y<f ( x ))
[ ∇ L ( f ( x ) , y )] (cid:12)(cid:12) , 0 < η < 1 be probability when 0 ≤ (cid:15) s , and 0 < ξ < 1 be probability when (cid:15) a = 0 .
Then, ∇ ˇ L ( f t ) is not consistent with the gradient in Equation 6 a.s., and the difference (bias) between them at step t + 1 in the gradient descent is
[Equation] [Equation]
Proof.
We outline a proof here and provide a complete one in Appendix A.3.
We ﬁrst show that the bias |∇ ˇ L ( f t ) − ∇L ( f t ) | can be represented by the difference between the expectation of g ( f t ( x ))
[Figure]
[Figure]
[Figure]
[Figure]
with the upper-side data and that with the lower-side data, which can be written by δ .
The bias also has the coefﬁcient which contains the proportions for the lower-side data and the original upper-side data mixed into the lower side due to incomplete observations.
These values can be written by η and ξ from their deﬁnitions.
Lemma 3.4 shows that the bias caused by incomplete observations becomes severe when there is a large difference between the expectations of the gradients in the upper side and the lower side.
δ is usually higher than zero because δ = 0 implies there is no difference between the expectations of the gradients in the upper side and the lower side or both of the expectations are zero.
Furthermore, a larger 1 − ξ = p ( (cid:15) a < 0) makes the bias more signiﬁcant, which agrees with the intuition that as the proportion of incomplete observations increases, the problem becomes more difﬁcult.
E XPERIMENTS
We now evaluate the proposed method through numerical experiments.
We ﬁrst introduce the baselines to be compared in the experiments.
Then, we present the experimental results to show the effectiveness of our unbiased learning.
Baselines.
Recall that the novelty of the proposed approach lies in the unbiased gradient in Equation 10, which is derived from the new class of loss functions in equation 9 with Condition 3.1.
An objective of our experiments is thus to validate the effectiveness of this new class of loss functions and the corresponding gradients against common loss functions in the literature.
Speciﬁcally, we compare the proposed method with MSE (mean squared error), MAE (mean absolute error), and Huber losses (Huber et al., 1964; Narula & Wellington, 1982; Wilcox, 1997).
For robust loss function in regression, MAE and Huber losses are considered the de facto standard and state-of-the-art in many studies and libraries.
We use the same model and optimization method with all of the loss functions under consideration, and hence the only difference among the proposed method and the baselines is in the gradients.
Since the loss function uniquely determines the baseline, we refer to each baseline method as MSE , MAE , or Huber .
4. E XPERIMENTAL P ROCEDURE AND R ESULTS
The experiments are organized into three parts.
In Section 4.1.1, we visually demonstrate the effectiveness of the proposed approach in giving unbiased prediction.
In Section 4.1.2, we intensively and quantitatively evaluate the predictive error of the proposed method and baselines with ﬁve real-world regression tasks.
In Section 4.1.3, we demonstrate the practical beneﬁt of our approach in a real healthcare use case, which has motivated this work.
See the appendix for the details of the experimental settings.
4.1. D EMONSTRATION OF U NBIASED L EARNING
Procedure.
We start by conducting the experiments with synthetic data to show the effectiveness of our method in obtaining unbiased learning results from asymmetrically-corrupted data with
[Table] [Table]
different proportions of incomplete observations, K = { 25 , 50 , 75 } % .
We use three synthetic tasks,
LowNoise , HighNoise
[Table]
and Breathing collected from the Kaggle dataset (Sen, 2016).
We compare the proposed method against MSE , which assumes that both upper- and lower-side data are correctly labeled.
This comparison shows whether our method can learn from asymmetrically-corrupted data in an unbiased manner, which MSE cannot do.
Results.
In Fig.
[Table]
we plot the error in prediction (i.e., the predicted value minus the true value) given by the proposed method and MSE for each data-point of the three tasks with K = 50% .
Note that, for evaluating the unbiasedness, these test sets do not have incomplete observations.
Since MSE regards both upper- and lower-side data as correctly labeled, it produces biased results due to the incomplete observations, where the average error (shown by the green dashed line) is negative, which means the estimation has a negative bias.
In contrast, the average error by the proposed method (shown by the blue solid line) is approximately zero.
This clearly shows that the proposed method obtained unbiased learning results.
The ﬁgures for the other settings and tables showing quantitative performance are in Appendix E.
4.1. P ERFORMANCE C OMPARISON A MONG D IFFERENT L OSS F UNCTIONS
Procedure.
We next apply the proposed method and baselines to ﬁve different real-world healthcare tasks from the UCI Machine Learning Repository (Velloso, 2013; Velloso et al., 2013) to show a more extensive comparison between the proposed method and the baselines ( MSE , MAE , and
Huber ).
For the proposed method, we use two implementations of L ( f ( x ) , y ) for f ( x ) ≤ y (cid:48) in Equation 10: the absolute loss ( Proposed-1 ) and the squared loss ( Proposed-2 ).
Here, we report the mean absolute error (MAE), and its standard error, of the predictions ˆ y =
{ ˆ y n } N n = against the corresponding true labels y
across 5 -fold cross-validation, each with a different randomly sampled training-testing split.
MAE is the common metric used in the healthcare domain (Lee et al., 2016; Yeung et al., 2002; Wang et al., 2005; Srinivas et al., 2020) and is deﬁned as MAE( y , ˆ y ) ≡ 1 / N (cid:80)
Nn =1 | y n − ˆ y n | .
For each fold of the cross-validation, we use a randomly sampled 20% of the training set as a validation set to choose the best hyperparameters for each algorithm, in which hyperparameters providing the highest MAE in the validation set are chosen.
Results.
As seen in Table 1, Proposed-1 and Proposed-2 largely outperformes the baselines.
The robust regression methods ( MAE and Huber ) did not improve in performance against MSE .
In particular, Proposed-1 and Proposed-2 respectively reduced the MAE by more than 20% and 30% on average, compared with baselines.
4.1. R EAL U SE C ASE FOR H EALTHCARE
Procedure Finally, we demonstrate the practicality of our approach in a real use case in healthcare.
From non-intrusive bed sensors installed under each of the four legs of a bed, we estimate the motion intensity of a subject that could be measured accurately but intrusively with ActiGraph, a gold standard sensor wrapped around the wrist (Tryon, 2013; Mullaney et al., 1980; Webster et al., 1982;
Cole et al., 1992).
If we can mimic the outputs from ActiGraph with outputs from the bed sensors, we can measure the motion with high accuracy and high coverage, while also easing the burden on the subject.
We divide the dataset into three pieces and evaluate the results with 3 -fold cross-validation.
We here use the evaluation metrics that are speciﬁcally designed for sleep-wake discrimination (Cole et al., 1992) i.e., proportion of correct prediction period and rate of false prediction.
Results.
Table 2 shows the proportion of correct prediction period and rate of false prediction, which indicate that the proposed method captured 89 percent of the total time period of the motions that were captured by ActiGraph, and false detection due to factors such as ﬂoor vibration was only 1 . 6 percent.
Furthermore, the proposed method captured 15 additional motions that were not captured by ActiGraph.
The baseline method MSE was severely underﬁtted, and most of the weights were zero; thus, we omitted these results.
Overall, our ﬁndings here demonstrate that ActiGraph can be replaced with bed sensors, and we can also use the bed sensors for the inputs of certain ActiGraph functions, such as sleep-wake discrimination (Cole et al., 1992).
See also Appendix G for further details, including the actual estimation results of the motion intensity.
D ISCUSSION
Limitations.
In this paper, we do not address symmetric label corruption, such as ordinary outliers, where the coverage and incompleteness are consistent between a label and explanatory variables.
Other established approaches can handle such cases.
Only when the corruption is asymmetric does it lead to the technical challenge we address here.
In that sense, we can handle the opposite asymmetric corruption, in which labels for some observations may become inconsistently higher than those for typical observations.
This can be handled as learning from lower-side labeled data and unlabeled data , i.e., LU regression.
Since our derivation of U2 regression is straightforwardly applicable to this LU regression case, we show only its learning algorithm in Appendix C.
Asymmetric Label Corruption in Classiﬁcation.
In the classiﬁcation problem setting, asymmetric label corruption is addressed with positive-unlabeled (PU) learning, where it is assumed that negative data cannot be obtained, but unlabeled data are available as well as positive data (Denis, 1998; De Comit ´ e et al., 1999; Letouzey et al., 2000; Shi et al., 2018; Kato et al., 2019; Sakai &
Shimizu, 2019; Li et al., 2019; Zhang et al., 2019; 2020; Chen et al., 2020b;a; Luo et al., 2021; Hu et al., 2021; Li et al., 2021).
An unbiased risk estimator has also been proposed (Du Plessis et al., 2014; 2015).
However, PU classiﬁcation cannot be used for a regression problem, where labels are real values and we need to handle order and gradation between labels.
This is because its derivation and algorithm are based on the nature that labels must be binary, i.e., only positive or negative.
We overcome this limitation with a novel approach based on an unbiased gradient.
Future work.
We showed that our approach to estimating hyperparameters based on the grid search with the validation set was effective even for the one contains the important ratio for upper-side labeled data, p ( f t ( x ) ≤ y ) .
It also provides the ﬂexibility needed to handle data variation.
Most studies on PU learning assume that a hyperparameter corresponding to ρ is given (Hammoudeh & Lowd, 2020; Sonntag et al., 2021; Lin et al., 2022), and some papers have addressed this hyperparameter estimation as their main contribution (Jain et al., 2016; Ramaswamy et al., 2016; Christoffel et al., 2016; Jain et al., 2020; Yao et al., 2021).
Developing a method for the hyperparameter estimation to improve performance would be a worthwhile next step of our study.
Also, in Assumption 2.1, we assumed (cid:15) s ⊥⊥ f ∗ ( x ) and (cid:15) a ⊥⊥ f ∗ ( x ) , which is a common noise assumption.
Addressing the case when the noises are not independent of f ∗ ( x ) is another future direction of our work.
Conclusion.
We formulated a regression problem from asymmetrically-corrupted data in which training data are corrupted with an asymmetric noise that always has a negative value.
This causes labels for data with relatively lower label values to be particularly unreliable.
To address this problem, we proposed a learning algorithm, U2 regression.
Under some technical assumptions, we showed that our algorithm is unbiased and consistent with regression that uses uncorrupted data without incomplete observations.
Our analysis is based on the equivalence of the gradient between them.
An experimental evaluation demonstrated that the proposed method was signiﬁcantly better than the methods without the assumption of the asymmetrical label corruption.
A P ROOFS A. P ROOF OF L EMMA 2.
Proof.
For the proof of Lemma 2.
we will derive two important lemmas from Assumption 2.1.
Then, we will prove Lemma 2.2 by using them.
We ﬁrst show f ∗ ( x ) ≤ y (cid:48) ⇒ (cid:15) a = 0 .
When f ∗ ( x ) ≤ y (cid:48) , we have from Equation 1 and Equation 4:
[Equation]
[Equation]
Since (cid:15) a ≤ 0 by Assumption 2.
we have | (cid:15)
[Equation]
(cid:15)
[Equation]
If (cid:15) a <
Assumption 2.1 implies | (cid:15) s | < | (cid:15) a | which contradicts Equation 13.
Hence, we must have (cid:15) a = 0 .
Since y = y (cid:48) when (cid:15) a = 0 , we have
[Equation]
which establishes
Lemma A.1.
Let p ( x , y, y (cid:48) ) be the underlying probability distribution for x , y , and y (cid:48) .
Then,
[Equation]
The condition parts of these conditional distributions represent the relationships between labels and regression functions, e.g., p ( x , y | f ∗ ( x ) ≤ y ) is the distribution of x and y when y is higher than what is given by the oracle regression function f ∗ .
Similar to Lemma A.1, we show f ( x ) ≤ y (cid:48) ⇒ (cid:15) a = 0 .
Let F (cid:48) ≡ { f ∈ F : | f ( x ) − f ∗ ( x ) | ≤
| (cid:15) s | a.s. } , which represents our natural expectation that the regression function f well approximates f ∗ .
When f ( x ) ≤ y (cid:48) , we have from Equation 1 and Equation 4 with the condition f ∈ F (cid:48) :
[Equation] [Equation]
Since (cid:15) a ≤ 0 by Assumption 2.1, we have
[Equation]
If (cid:15) a < 0 , Assumption 2.1 implies 2 | (cid:15) s | < | (cid:15) a | , which contradicts Equation 18.
Hence, we must have
[Equation]
Since y = y (cid:48) when (cid:15) a = 0 , by replacing f ∗ with f for the argument in the derivation of Lemma A. in Equation 15, we have
Lemma A.2.
Let F (cid:48) ≡ { f ∈ F : | f ( x ) − f ∗ ( x ) | ≤
| (cid:15) s |} .
When f ∈ F (cid:48) , the following holds:
[Equation]
Lemma A.1 immediately implies
[Equation]
for any function G : R D × R → R as long as the expectations exist.
When f ∈ F (cid:48) , from Lemma A.2, we then have
[Equation]
A. P ROOF OF P ROPOSITION 3.
Proof.
From the decomposed gradients ∇L ( f t ) in Equation 7, we derive the proposed gradient only with the expectations over p ( x , y (cid:48) ) .
From Condition 3.1 for L ( f ( x ) , y ) , ∇ L ( f ( x ) , y ) =
g ( f ( x )) when y < f ( x ) .
Thus, Equation 7 can be rewritten as
[Equation]
where y is marginalized out in the expectation in the second term since g ( f t ( x )) does not depend on y .
Here, Equation 6 and Equation 7 can be rewritten by replacing ∇ L ( f t ( x ) , y ) with g ( f t ( x )) , as
[Equation]
Since g ( f t ( x )) does not depend on y
[Equation]
we can marginalize out y in Equation 25 as
[Equation] [Equation]
From Equation
[Equation]
we can express Equation 23 as
[Equation]
Finally, from Lemma 2.2, we can rewrite Equation 27 as:
[Equation]
which is identical to Equation 9.
Thus, the gradient in Equation 9 is unbiased and consistent with the gradient in Equation 6 a.s.
A. P ROOF OF L EMMA 3.
Proof.
The difference between the decomposed gradients ∇ ˇ L ( f t ) and ∇L ( f t ) at step t + 1 in the gradient descent is
[Equation]
From Lemma
[Equation]
2 and Condition
[Equation] [Equation]
We decompose E p ( x | y (cid:48) <
f t ( x ))
[ g ( f t ( x ))] again as
[Equation] [Equation]
The condition y (cid:48) <
f t ( x ) ∧ y <
f t ( x ) is equivalent to the condition y <
f t ( x ) since y (cid:48) ≤ y from
Assumption 2.1 and thus p ( y (cid:48) <
f t ( x ) | y <
f t ( x ))
= 1 .
Then, we have
[Equation]
t ( x ))
E p ( x | y (cid:48) <
f t ( x ) ∧ f t ( x ) ≤ y )
[ g ( f t ( x ))]
+ p
( y < f t ( x ) | y (cid:48) <
f t ( x ))
E p ( x | y<
f t ( x ))
[ g ( f t ( x ))]
(cid:19) − p ( y < f t ( x ))
E p ( x | y<
f t ( x ))
[ g ( f t ( x ))] (cid:12)(cid:12)(cid:12)(cid:12) .
Additionally, since p ( y < f t ( x ) | y (cid:48) <
f t ( x ))
= 1 − p ( f t ( x ) ≤ y | y (cid:48) <
f t ( x )) , |∇ ˇ L ( f t ) − ∇L ( f t ) | (33) = (cid:12)(cid:12)(cid:12)(cid:12) p ( y < f t ( x )) p ( f t ( x ) ≤ y | y (cid:48) <
f t ( x )) (cid:18)
E p ( x | y (cid:48) <
f t ( x ) ∧ f t ( x ) ≤ y )
[ g ( f t ( x ))] − E p ( x | y<
f t ( x ))
[ g ( f t ( x ))]
(cid:19)(cid:12)(cid:12)(cid:12)(cid:12) .
This equation shows that the bias is represented by the difference between the expectation of g ( f t ( x )) with the lower-side data and that with the original upper-side data mixed into the lower side due to incomplete observations and the corresponding proportions.
From Assumption 3.3, since (cid:15) a ⊥⊥ x ,
[Equation]
( f
[Equation] [Equation]
Since | f − f ∗ | ≤ | (cid:15) s | a.s., p ( f t ( x ) ≤ y )
=
η and p ( y < f t ( x ))
= 1 − η from their deﬁnition, p ( f t ( x
[Equation] [Equation]
Therefore, from the deﬁnition of δ ,
[Equation]
B I MPLEMENTATION OF L EARNING A LGORITHM B ASED ON S TOCHASTIC O PTIMIZATION
We scale up our U2 regression algorithm by stochastic approximation with M mini-batches and add a regularization term, R ( f ) :
[Equation]
where ∇ ˆ L { m } ( f t ) is the gradient for the m -th mini-batch, { X { m } up , y { m } up } and X { m } un respectively are upper-side and unlabeled sets in the m -th mini-batch based on the current f t , λ is a regularization parameter, and the regularization term R ( f ) is, for example, the L 1 or L 2 norm of the parameter vector θ of f .
We also convert n up / ( π up N ) to a hyperparameter ρ , ignoring constant coefﬁcients instead of directly handling π up .
The hyperparameters ρ and λ are optimized in training based on the grid-search with the validation set.
The U2 regression algorithm based on stochastic optimization is described in Algorithm 1.
We learn the regression function with the gradient in Equation 37 by using any stochastic gradient method.
Here, we used Adam with the hyperparameters recommended in Kingma & Ba (2015), and the number of samples in the mini-batches was set to 32 .
We set the candidates of the hyperparameters, ρ and λ , to { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
By using the learned f , we can estimate ˆ y =
f
( x ) for new data x .
C A LGORITHM FOR LU R EGRESSION
We show the algorithm for the lower and unlabeled regression (LU regression), where labels for some observations may become inconsistently higher than those for typical observations.
Let L LU ( f ( x ) , y ) be a loss function for LU regression and g LU ( f ( x )) be a gradient ∇ L LU ( f ( x ) , y ) when f ( x ) ≤ y .
Similar to Condition 3.1 for U2 regression, we assume that the class of L LU ( f ( x ) , y ) satisﬁes the condition that g LU ( f ( x )) is a gradient function depending only on f ( x ) and not on the value of y .
Then, LU regression is Algorithm 1, with the following gradient, ∇ ˆ L { m } LU ( f t ) , instead of ∇ ˆ L { m } ( f t ) in Equation 37, as
[Equation]
where { X { m } lo , y { m } lo } and X { m } un respectively are lower-side and unlabeled sets in the m -th minibatch based on the current f t .
D C OMPUTING INFRASTRUCTURE
All of the experiments were carried out with a Python and TensorFlow implementation on workstations having 80 GB of memory, a 4 . 0 GHz CPU, and an Nvidia Titan X GPU.
In this environment, the computational time to produce the results was a few hours.
E D ETAILS OF EXPERIMENTS IN S ECTION 4.1. E. S YNTHETIC DATASETS
We conducted the experiments on synthetic data to evaluate the feasibility of our method for obtaining unbiased learning results from asymmetrically-corrupted data containing different proportions of incomplete observations.
We generated synthetic data on the basis of Assumption 2.1 and Equation 4.
We randomly generated N = 1000 training samples, X = { x n } Nn =1 , from the standard Gaussian distribution N ( x n ; 0 , I ) , where the number of features in x was D = 10 , and I is the identity matrix.
Then, using X , we generated the corresponding N sets of true labels y =
{ y n } Nn =1 from the distribution N ( y n ; w (cid:62) x n , β ) , where w are coefﬁcients that were also randomly generated from the standard Gaussian distribution N ( w ; 0 , I ) , β is the noise precision, and (cid:62) denotes the transpose.
For simulating the situation in which a label has incomplete observations, we created corrupted labels y (cid:48)
= { y (cid:48) n } Nn =1 by randomly selecting K percent of data in y and subtracting the absolute value of white Gaussian noise with twice the value of the precision as that of y , 2 β , from their values.
We repeatedly evaluated the proposed method for each of the following settings.
The noise precision was β = { 10 0 , 10 − 1 } , which corresponded to a low-noise setting task ( LowNoise ) and a high-noise setting task ( HighNoise ), and the proportion of incomplete training samples was K = { 25 , 50 , 75 } % .
In the case of K = 75% , only 25 percent of the samples correctly corresponded to labels, and all of the other samples were attached with labels that were lower than the corresponding true values.
It is quite difﬁcult to learn regression functions using such data.
In these tasks, we used a linear model, θ (cid:62) x , for f ( x ) and an implementation for Equation 37 with the absolute loss, which satisﬁes
Condition 3.1, for the loss function L and L 1 -regularization for the regularization term.
We set the candidates of the hyperparameters, ρ and λ , to { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
We standardized the data by subtracting their mean and dividing by their standard deviation in the training split.
We used Adam
[Figure]
MSE
[Figure]
× 0 -1- -3-
[Figure]
with the hyperparameters recommended in Kingma & Ba (2015), and the number of samples in the mini-batches was set to 32 .
We also used a real-world sensor dataset collected from the Kaggle dataset (Sen, 2016) that contains breathing signals ( Breathing ).
The dataset consisted of N = 1 , 432 samples.
We used signals from a chest belt as X = { x n } Nn =1 , and x in each sample had D = 2 number of features, i.e., the period and height of the expansion/contraction of the chest.
We used signals obtained by the Douglas bag (DB) method, which is the gold standard for measuring ventilation, as true labels y
=
{ y n } Nn =1 .
For our problem setting, we created corrupted labels y (cid:48)
= { y (cid:48) n } Nn =1 through the same procedure for synthetic corruption as that for LowNoise and HighNoise with K = { 25 , 50 , 75 } % .
In the experiment on Breathing, for its non-linearity, we used θ (cid:62) φ ( x , σ ) for f ( x ) , where φ is a radial basis function with the training set as its bases, and σ is a hyperparameter representing the kernel width that is also optimized by using the validation set.
We set the candidates of the hyperparameter σ to { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
The other implementation details were the same as those for LowNoise and HighNoise .
E. D ETAILED RESULTS
Figure 3 shows the error between the estimation results of the proposed method and their true values and those of MSE for LowNoise , HighNoise , and Breathing with 25 and 75 percent of incomplete training samples.
Table 3 shows the performance on LowNoise , HighNoise , and Breathing for the proposed method and MSE .
As shown in Figure 3, the proposed method obtains unbiased learning results in all cases, while MSE produces biased results.
From Table 3, we can see that the proposed method outperformes MSE overall.
We found that the performance of our method is not signiﬁcantly affected by the increase in the proportion of incomplete training samples K even for K = 75% , unlike that of MSE .
F D ETAILS OF EXPERIMENTS IN S ECTION 4.1.
We applied the algorithm to ﬁve different real-world healthcare tasks recorded in the datasets from the UCI Machine Learning Repository (Velloso, 2013; Velloso et al., 2013), which contains sensor outputs from wearable devices attached to the arm while subjects exercised.
From the non-intrusive
[Table]
sensors attached to gym equipment, we estimated the motion intensity of a subject that was measured accurately with an arm sensor that was an intrusive sensor wrapped around the arm.
If we can mimic outputs from the arm sensor with outputs from the equipment sensor, it could contribute to the subjects’ comfort, as they would not need to wear sensors to measure their motion intensity.
We used all of the features from the equipment sensor that took “None” values less than ten times as X = { x n } Nn =1 , where each sample had D = 13 number of features.
The corrupted labels y (cid:48)
= { y (cid:48) n } Nn =1 were the magnitude of acceleration from the arm sensor, which can accurately sense motion intensity on the arm, but it had insufﬁcient data coverage and incomplete or missing observations for the movements of other body parts.
For performance evaluation, we used the magnitude of acceleration for the entire body as true labels y =
{ y n } Nn =1 .
The number of samples were N = 11 , 159 , N = 7 , 593 ,
N = 6 , 844 , N = 6 , 432 , and N = 7 , 214 respectively for the tasks, Speciﬁcation, Throwing A,
Lifting, Lowering, and Throwing B.
For the complex nature of the tasks, we used a 6 -layer multilayer perceptron with ReLU (Nair & Hinton, 2010)
(more speciﬁcally, D - 100 - 100 - 100 - 100 - 1 ) as f ( x ) , which demonstrates the usefulness of the proposed method for training deep neural networks.
We also used a dropout (Srivastava et al., 2014) with a rate of 50% after each fully connected layer.
We used two implementations for L ( f ( x ) , y ) when f ( x ) ≤ y (cid:48) in Equation 37 with the absolute loss ( Proposed-1 ) and the squared loss ( Proposed-2 ).
For both implementations, we used the absolute loss, which satisﬁes Condition 3.1, for the loss function L ( f ( x ) , y )
when y (cid:48) < f ( x ) and used L 1 -regularization for the regularization term.
The other implementation details were the same as those for LowNoise , HighNoise , and Breathing .
G D ETAILS OF EXPERIMENTS IN S ECTION 4.1.
We demonstrate the practicality of our approach in a real use case in healthcare.
From non-intrusive bed sensors installed under each of the four legs of a bed, we estimated the motion intensity of a subject that was measured accurately with ActiGraph, a gold standard intrusive sensor wrapped around the wrist (Tryon, 2013; Mullaney et al., 1980; Webster et al., 1982; Cole et al., 1992).
The sensing results of ActiGraph are used for tasks such as discriminating whether a subject is asleep or awake (Cole et al., 1992).
While ActiGraph can accurately sense motion on the forearm, it has insufﬁcient data coverage in other areas and often causes observations of movements on other body parts to be missing.
The bed sensors have a broader data coverage since they can sense global motion on all body parts; however, the sensing accuracy is limited due to their non-intrusiveness.
If we can mimic the outputs from ActiGraph with outputs from the bed sensors, we can expect to achieve sufﬁcient accuracy and coverage while also easing the burden on the subject.
The dataset we used included three pieces of data, Data (i), (ii), and (iii), which were respectively recorded over 20 , 18 , and 18 .
5 minutes.
Each piece of data consisted of pairs of bed-sensor-data sequences and the corresponding motion intensity sequence obtained by ActiGraph.
We used the “magnitude” attribute of ActiGraph as corrupted labels y (cid:48) for the motion intensity, whose sampling rate was about one sample per second.
For true labels y , we manually measured the motion intensity every minute under the management of a domain expert.
For X , we ﬁrst computed the gravity center of the four sensor outputs that were obtained from the bed sensors under the four legs of a bed.
Then, we computed the time derivatives and cross terms of the raw sensor outputs and the gravity center.
The sampling rate of the bed sensors was different from that of ActiGraph, about one sample per ﬁve
[Figure]
milliseconds.
Thus, X was ﬁnally generated as a sliding window of statistics in 1 , 000 -millisecond ( 1 second) subsequences of the time series of the above computed variables, where 1 second was the same as the sampling interval of ActiGraph.
The statistics were means, standard deviations, and { 0 . 05 , 0 . 25 , 0 . 5 , 0 .
95 } quantiles.
In this task, we used the linear model θ (cid:62) x for f ( x ) due to its interpretability, which is inevitable in real-world healthcare and medical applications.
G. E STIMATION RESULTS FOR MOTION INTENSITY
Figure 4 compares our estimation results for motion intensity with the output of ActiGraph and true labels.
G. I MPORTANT FEATURES ESTIMATING MOTION INTENSITY
The important features selected by L 1 regularization were the statistics of the gravity center and the cross terms and time derivatives of the raw sensor outputs.
The largest weight was assigned to the standard deviation of the gravity center, which represents the amplitude of the gravity center, so it is directly related to the motion of subjects.
H O THER P OSSIBLE U SE C ASES OF R EGRESSION FOR S ENSOR M AGNITUDE
Examples of predicting the magnitude values of a sensor, which is a ﬁeld of application of U regression, can be found in several areas.
Besides the medical and healthcare applications discussed in the main text, another example is estimating the wind speed or rainfall in a speciﬁc region from observable macroscopic information (Cheng & Tan, 2008; Abraham & Tan, 2010; Abraham et al., 2013; Vandal et al., 2017), known as statistical downscaling (Wilby et al., 2004).
Wind speed and rainfall, which are labels in these tasks, can be sensed locally in a limited number of locations and provide incomplete observations and biased labels compared with macroscopic information, which is considered to be explanatory variables.