N OISE + 2N OISE : C O - TAUGHT D ENOISING A UTOENCODERS FOR T IME -S ERIES D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the task of learning to recover clean signals given only access tonoisy data.
Recent work in computer vision has addressed this problem in thecontext of images using denoising autoencoders (DAEs).
However, to date DAEsfor learning from noisy data have not been explored in the context of time-seriesdata.
DAEs for denoising images often rely on assumptions unlikely to hold in thecontext of time-series, e.g. , multiple noisy samples of the same example.
Here,we adapt DAEs to cleaning time-series data with noisy samples only.
To recoverthe clean target signal when only given access to noisy target data, we leverage anoise-free auxiliary time-series signal that is related to the target signal.
In addition to leveraging the relationship between the target signal and auxiliary signal,we iteratively filter and learn from clean samples using an approach based on coteaching.
Applied to the task of recovering carbohydrate values for blood glucosemanagement, our approach reduces noise (MSE) in patient-reported carbohydratesfrom 72 g 2 (95% CI: 54,93) to 18 g 2 (13,25), outperforming the best baseline (MSE= 33 g 2 (27,43)).
We demonstrate strong time-series denoising performance, extending the applicability of DAEs to a previously under-explored setting.
I NTRODUCTION
Background.
Denoising autoencoders (DAEs) (Vincent et al., 2008) have been used to accuratelydenoise various signals, including medical images (Gondara, 2016), ECG signals (Xiong et al.,2016), and power system measurements (Lin et al., 2019).
With respect to time-series data, DAEshave been used for forecasting (Romeu et al., 2015), classification (Zheng et al., 2022) and imputation (Zhang & Yin, 2019), but generally require access to clean samples at training and do notprovide de-noised outputs.
In many real-world settings, clean samples are unavailable at training.
Work in computer vision has addressed this problem through extensions that either require pairedsamples (Lehtinen et al., 2018) or rely on patch-based analysis (Krull et al., 2018; Laine et al., 2019;
Xie et al., 2020; Batson & Royer, 2019).
Similar approaches do not extend to time-series data,where paired samples rarely exist and patch-based techniques do not apply.
Beyond approaches thatrely on paired samples or patch-based analyses, researchers have recently proposed techniques thatutilize knowledge of the noise distribution to recover the clean signal.
These approaches either usethe properties of the distribution to recover the clean signal after training on noisy data (Kim & Ye,2021; Moran et al., 2019), or rely on the noise having low expectation and variance compared to thesignal, in which case a model trained on noisy data can approximate one trained on clean data (Xuet al., 2020).
While these approaches may be considered in a time-series setting (and are treated asbaselines here), their applicability is limited as noise in time-series settings is rarely weak or known.
Our Contribution.
In light of this gap, we adapt denoising autoencoders for time-series data.
Ourapproach, ‘Noise + 2Noise’, learns to map a noisy target signal to a clean signal given only noisysamples and an auxiliary clean signal.
Inspired by work in image denoising (Lehtinen et al., 2018;
Xu et al., 2020), we add additional noise to the noisy target signal during training and attempt torecover the original noisy signal.
Provided that the noise has low expectation and variance, a networktrained in this manner can learn to recover the true signal because the noise will minimally impactthe expected value of the output of the network (Xu et al., 2020).
The auxiliary signal is inputalong with the target signal into a denoising autoencoder, which allows our network to leverage therelationship between the auxiliary and target signals.
To address the fact that the signal to noiseratio might not be weak, we adapt a co-teaching approach to train two DAEs (Jiang et al., 2018; Han et al., 2018).
We use this approach to identify the cleaner samples; the most likely low-noisesamples are identified as the low-loss samples of the other model and used for backpropagation.
This co-teaching approach has never been applied to de-noising in a time-series or any other setting.
It has also never been utilized in a continuous output setting.
By adapting this co-teaching approachto DAEs, we provide a solution to denoising in this novel (time-series) setting.
Real-world Inspiration.
Disparate levels of noise across variables are common in data streams.
Measurement reliability can vary across sensors, from essentially noiseless to highly corrupted.
Throughout this work we take inspiration from a real-world problem affecting millions in the US:blood glucose management.
Individuals with diabetes monitor several variables over time, includingtheir blood glucose, insulin administrations, and carbohydrate intake.
Blood glucose when measuredby a continuous glucose monitor (CGM) has little noise, while the number of grams of carbohydratesin a meal are patient-reported, and, as a result, are often inaccurate.
Recognizing the variation in thelevel of noise across signals in a data stream, we propose an approach that utilizes the more reliablevariables ( e.g. , blood glucose) to update noisy variables like carbohydrates.
Our approach aims toretrospectively correct measurements of patient-reported signals, like carbohydrates in the contextof diabetes, which could lead to improved treatment decisions: patients can learn when they areover- or under-reporting and adjust in the future, ultimately improving health outcomes.
R ELATED W ORK
In denoising autoencoder training, a model input is corrupted and a network is tasked with recoveringthe original input.
In this way, the network cannot learn the identity, unlike in basic autoencodertraining (Vincent et al., 2008).
Recent work in machine learning has focused on using DAEs torecover clean signals from only noisy signals.
The vast majority of this work lies in image analysisand builds off of “noise2noise” (Lehtinen et al., 2018), an approach that uses multiple noisy instancesof the same image to train a model to learn to denoise the image.
Briefly, the approach relies on thefact that if the noise is mean zero, using a secondary noisy instance (besides the input image) asa target will produce a network that learns the clean image, in expectation, when enough trainingdata is available.
When paired samples are unavailable, others have proposed approaches basedon exploiting patches sampled from the image (Krull et al., 2018; Laine et al., 2019; Xie et al. ,2020; Batson & Royer, 2019), but these are not applicable to time-series data.
Other approacheseschew relying on inter-variable relationships but rely heavily on a known noise function (Moranet al., 2019; Kim & Ye, 2021) or a low expectation and variance noise function (Xu et al., 2020).
Our approach builds off of Xu et al.
(2020), learning to reconstruct a noisy signal from only noisysamples of that signal, but in contrast to Moran et al.
or Kim & Ye (2021), we do not makestrong assumptions about the noise distribution.
To get around these assumptions, we leverage anauxiliary signal and iteratively identify the cleaner samples within the dataset.
Combined with theauxiliary signal these samples are used to train a DAE that treats these samples as ground truth.
Our approach to denoising a noisy target signal by iteratively identifying clean samples is, in part,related to work in noisy-label learning.
A common approach to learning from noisy labels involvesidentifying and reweighting samples with clean labels during training.
Samples are filtered basedon gradient values (Ren et al., 2020), Jacobian ranking(Mirzasoleiman et al., 2020) or some latentstate (Lee et al., 2019; Wu et al., 2020).
Co-teaching (Han et al., 2018), which builds off of mentornet (Jiang et al., 2018), is performed by training two networks in parallel.
Each network is backpropgated using only samples within the current mini-batch for which the loss of the other networkis lowest.
Intuitively, samples with incorrect labels are likely to have higher loss and therefore beremoved.
Using two networks in parallel provides robustness to outliers and initially misclassifiedsamples, which single-network boosting-style approaches are sensitive to.
These approaches havebeen primarily explored in a supervised setting.
In contrast, we consider an unsupervised setting inwhich labels are unavailable and, instead, the input signals themselves are corrupted.
To the best ofour knowledge, such a co-teaching approach has not been explored in the context of denoising.
P ROBLEM S ETUP
Problem Definition.
Given a noisy target variable and a reliably measured auxiliary time-series, weaim to recover the true values of the target variable.
We assume a relationship between the auxiliary and target variables, and that some samples from the noise distribution associated with the targetvariable will be close to zero, although which samples is unknown in advance.
Formalization.
Let x ∈ R denote a sample of the target variable.
Let n ∼ N , where n ∈ Rdenotes a random variable drawn from an unknown distribution N .
Let b ∈ R T denote an auxiliarytime-series that is related to x , such that a mapping f ( b ) → x exists.
Given a sample y = x + n andb , we aim to learn to recover x .
Assumptions.
We assume that the distribution N is independent of both b and Y .
We also assumethat some values of n are near zero such that given training data with k samples { y i } ki =1 with noisevalues { n i } ki =1 and true signal values { x i } ki =1 , there exists a subset S with sufficient size for trainingsuch that the mean and variance of n s ∀ s ∈ S are negligible compared to the mean and variance ofx s ∀ s ∈ S .
We assume that the distribution of low noise samples is such that they cover all regionsof the input; i.e. that S does not exclude entire regions of Y .
Finally, we assume that the relationshipbetween b and x can be accurately captured with a recurrent neural network (RNN).
M ETHODS
Overview.
Our method, ‘Noise + 2Noise’ ( N + 2N ), is summarized in Figure 1 .
We filter out noisysamples during training, refining the model parameters on selected samples of y estimated to havethe least noise.
These filtered samples are used to learn to denoise the signal.
To identify thesamples within a batch to use during training, y and b are passed to a DAE that outputs ˜ y , or ade-noised y value.
Assuming the DAE accurately reconstructs x , the loss L (˜ y, y ) represents theexpected error between y and x .
Given a batch { y i } ni =1 , we identify the subset of samples withvalues of L (˜ y, y ) below the R ( t ) th percentile (where R ( t ) is an increasing function of iteration t ).
These ‘low-noise’ samples, y j ∀ j ∈ J , are then augmented with additional noise m j ∼ M and,along with corresponding b vectors, are input to the DAE, which outputs a reconstruction of y j :ˆ y j .
We then backpropagate using the squared error between ˆ y j and y j .
In this way, the DAE istrained on the samples estimated to have the least noise while utilizing the noisy signal as inputwithout the susceptibility to learning the identity function that comes with standard auto-encodertraining.
Co-teaching is utilized in that two DAEs are used at each step ( DAE 1 and DAE 2 ), andduring training, the samples identified as low-loss by DAE 1 are augmented and passed to DAE 2for backpropagation, and vice-versa.
This adds the benefit of ensembling and curtails sensitivity tooutliers or incorrectly selected samples.
We note that the auxiliary signal b is input to the modelduring both the sample selection and backpropagation steps.
This allows our approach to functioneven when the variable of interest y is too noisy or low-dimension to be de-noised alone.
Denoising Autoencoder.
In our setup, additional noise ( m ) is added to y to produce z , a ‘doubly’noisy measurement of x . z and b are input to a network (henceforth denoted DAE) that outputsˆ y = DAE ( z, b ) , and the network is trained to reconstruct y : loss is measured between y and ˆ y .
As shown by Xu et al.
(2020), when the expectation and variance of the noise distribution N arenegligible compared to those of the signal, the model parameters that minimize the loss between ˆ yand y are very close to the optimal parameters of a model trained on clean data.
Thus, provided thesignal to noise level is high enough, we can pass y to DAE and expect a reduction in noise, with anoutput much closer to x , at inference time.
Co-teaching DAEs.
We do not expect the noise to be weak in general, but we do assume thatsome of the samples will be lower noise than others.
We identify and train using these samplesvia an adapted co-teaching approach (Han et al., 2018).
We utilize two DAEs, and for each, webackpropagate using only the samples for which the denoised y values from the other DAE are nearthe original y values.
If the denoised y values approach x , we are then selecting samples for whichthe estimated noise n is lowest.
Claim.
When using co-teaching to train two DAEs ( DAE 1 and DAE 2 ) in parallel, ˜ y
=
DAE 1 ( y, b ) and ˜ y 2 = DAE 2 ( y, b ) approach x .
Justification.
Based on the main result ofXu et al.
(2020), these values approximately equal x whena DAE is trained on a dataset where the expectation and variance of the signal are much greater thanthose of the noise.
We have assumed that such a sub-sample exists in our dataset, and we hypothesizethat co-teaching will select such a sub-sample.
Note that because the noise n is independent of bothy and b , any signal learned by DAE 1 must be of the form DAE 1 ( y, b )
= g ( x ) + q , for somefunction g , where q is independent of all variables.
The DAE might begin to learn a biased functionof x ; for example, if the noise distribution N is not mean-zero, a DAE might learn the functiong ′ ( x ) = x + ¯ N .
We curtail this behavior by initializing the DAEs to output the identity function( y ).
Because the model is near the identity function g ( x ) =
y early in training, if it began to learna biased function such as g ′ , it would begin to output values
g ′ ( x ) ≈ g ′ ( y ) =
y + ¯ N .
These valueswould be further from the identity than if the model learned a non-biased function of x , which wouldin turn guide the model away from learning g ′ .
By similar logic, the model would be encouraged tolearn a non-biased function of x , the simplest of which is g ( x ) = x .
This is only a conjecture, butin practice, we have found that this approach works well even when there is fairly substantial biasin the noise.
We note that although the two DAEs are initialized to output the identity, their weightsare otherwise random so they are not likely to converge to the same minima.
We further preventconvergence by utilizing the co-teaching+ variant (see below).
Sample selection.
We select the samples with the lowest estimated noise to backpropagate with.
If ˜ y 1 and ˜ y 2 approach each network’s estimated value of x , then DAE 1 ’s estimate of n , the noisebetween x and y , is approximately y − ˜
y 1 (and similar for DAE 2 ).
For loss function L , we use
L (˜ y 1 , y ) and L (˜ y 2 , y ) to select samples.
Given a batch { y i } ni =1 , at iteration t , we identify thesubset of samples with values of L (˜ y {+
+} i , y i ) below the R ( t )
th percentile as J 1 =
{ j : L ( y j , ˜ y 1 j ) <
P R ( t ) ( L ( { y j } , { ˜ y 1 j } )) } , and similarly define J 2 for DAE 2 .
As in Han et al.
(2018), we begin bytraining on the full sample.
Over the course of training, as the DAEs are expected to become moreaccurate, we gradually reduce the sample.
This prevents the memorization of noisy samples that canoccur later in training.
Hyper-parameter τ ∈ (0 , 1) represents the maximum proportion of samplesremoved and E k represents the iteration at which we stop increasing the proportion of samplesremoved.
A linear decrease in sample size as a function of iteration t is implemented by using thelowest-loss R ( t ) =
(1 − Maximum ( tE K τ, τ )) · 100% of samples for backpropagation.
Training.
Each DAE is trained on the samples for which the other network estimates that the noiseis lowest: samples selected by DAE 1 ( y j ∀ j ∈ J 1 ) are augmented with additional noise m j ∼ Mto generate z j values.
z j , along with corresponding b vectors, are input to DAE 2 , which outputsa reconstruction of y j : ˆ y 2 j .
We then backpropagate using the squared error between ˆ y 2 j and y j .
Similarly, we only use samples y j ∀ j ∈ J 2 , augmented with m j ∼ M , to backpropagate DAE 1 .
By selecting samples based on L (˜ y, y ) rather than based on L (ˆ y, y ) , we are able to select a sampleindependent of secondary noise value m . Selecting samples dependent on m would be confoundingbecause samples might then be selected based on how low the value of m is at the current iteration,rather than the value of n , which is hidden.
Back-propagation is performed on an input that does not include clean y values, so the model is not likely to learn the identity function.
Sample selection isalways performed by the other DAE, so compared to boosting or other one-network approaches, ourmethod is less sensitive to error propagation from wrongly selected samples early in training.
Co-teaching+.
We utilize co-teaching plus (Yu et al.
where samples for which the modelsdisagree are selected for backpropagation
As a result, each model learns from the samples forwhich the other model’s estimates were better.
This also prevents the models from learning fromthe samples that they agree upon, which prevents convergence, maintaining unique strengths ineach model.
We remove the σ % of samples for which the models’ outputs are closest (here σ is ahyperparameter).
This step is performed prior to the sample selection step: the σ % of samples forwhich the distance between ˆ y 1 and ˆ y 2 are lowest are removed, and then the remaining samples forwhich L (˜ y 1 , y ) is lowest are used for the backpropagation of DAE 2 and vice versa.
R EAL - WORLD P ROBLEM S ETUP : B LOOD G LUCOSE M ANAGEMENT
To explore the benefit of our proposed approach, we consider a real-world problem setup basedon blood glucose management that inspired the setting described in Section 3.
Nearly two millionpeople in the US have type I diabetes and require insulin to maintain healthy glucose levels due toa pancreatic deficiency in insulin production.
Because of this, they must deliver boluses of insulinthrough an injection or an insulin pump prior to eating to counteract the rise in blood sugar that results from the ingestion of meals.
Bolus amounts are calculated based on patient-reported estimatesof carbohydrates.
Carbohydrates and bolus insulin generally cause blood glucose values to increaseor decrease after a delay of 30 minutes to an hour.
In our setup, carbohydrates correspond to xvalues, glucose levels are b .
Blood glucose forecasting and control have been extensively studied in the past (Silvia Oviedo,2016; Fox et al., 2020).
Accurate models for blood glucose dynamics are critical to the development of algorithms for managing blood glucose in individuals with diabetes both in terms ofpatient-selected treatment options and automated solutions.
Work in this field is popular largely because the ubiquity of devices for measuring blood glucose and administering insulin make obtainingfairly clean measurements for those values straightforward.
However, carbohydrates consumed arepatient-reported and as a result are often inaccurate.
This in turn leads to inaccurate doses of insulinand poor blood glucose management.
Besides misestimation, there are other sources of inconsistency between recorded carbohydrate values and their effects on blood glucose.
Variability in mealtypes is generally poorly captured, which is problematic because the effect of carbs on blood glucosecan be moderated by how quickly a meal was consumed, or the amount of protein, fat and other nutrients eaten.
Additionally, the timing of a meal may not be recorded accurately.
These factors alonemake utilizing carbohydrate information difficult, even when carbs are accurately recorded.
In anunsupervised setting, denoising approaches could learn representations of carbohydrate values thatincorporate these other sources of variability.
These representations could be more relevant to bloodglucose management than the exact number of grams consumed.
This could improve performanceof forecast and control algorithms.
E XPERIMENTAL S ETUP
We implement our approach in the context of learning to correct noisy patient-reported carbohydratemeasurements.
We compare performance to several baselines across real and simulated datasets.
6. D ATASETS
We utilize two T1D-based datasets.
The simulated dataset provides access to ground truth to whichwe can directly compare our method’s denoised outputs.
The real dataset provides a more challenging setting for quantifying the efficacy of our approach, but corresponds to real-world scenarios.
Both datasets are publicly available and have been previously explored in the context of forecastingand control (Man et al., 2014; Xie, 2018; Marling & Bunescu, 2018; 2020).
Both datasets consistof blood glucose, bolus (fast-acting) insulin, basal (slow-acting) insulin, and carbohydrate values.
All variables were scaled to be between zero and one.
For both datasets, time-series trajectories foreach patient were split into windows of 2 hour length ( T = 24 5-minute time points).
We ignore windows where a carbohydrate occurs in anywhere
but the first position, using only windows withno carbohydrates or carbohydrates at the beginning of the window during training.
This means wealso ignore windows with more than one carbohydrate present.
In a real-world setting these valuescould be updated recursively, but we simplify our setting here.
Simulated .
Our primary analyses are performed on data generated with a commonly-used T1Dsimulator.
We use the UVA-Padova simulator (Man et al., 2014) via a publicly available implementation (Xie, 2018).
For ten simulated individuals (the “adult” patients modeled in the simulator), wegenerated approximately 150 days worth of data each, in 30 day roll-outs of the simulator.
Dayswhere a patient either had more than 25 timepoints of glucose at the minimum value of 40, or morethan 35 timepoints over 450 were thrown out for being non-realistic.
The meal schedule used togenerate simulated data was based on the Harrison-Benedict equation (Harris & Benedict, 1919) asimplemented in (Fox et al., 2020).
In our simulation, for all datasets generated, we used the defaultbasal-bolus controller from the existing implementation of the simulator to administer insulin, butwe delayed five sixths (randomly selected) of the bolus administrations up to 3.5 hours, with delaytime randomly sampled from a uniform distribution.
The delay allows for disentanglement betweencarbohydrate and bolus effects.
The carbohydrate values serve as x values, while the CGM valuesoutput by the simulator serve as b values.
Noise was added to carbs during data generation.
We use noise proportional to the true carb value, as studies on the accuracy of carbe counting report errors relative to the total carbs consumed (Brazeau et al., 2012).
Also based on Brazeau et al.(2012), we use a noise distribution with a negative bias, as the carbs were found to be more-oftenunder-reported than not.
We therefore set y =
(1 + N ( − . 25 , .
x .
We then cap y below and aboveat 1 and 200 to keep values realistic.
We consider additional noise distributions as sensitivity analyses.
Bolus values were calculated based on the noisy carbohydrate values.
Additionally, 20% ofcarbohydrates are not reported, to make the dataset more realistic, as missingness is commonplace.
Real .
This dataset includes both the OHIOT1DM 2018 and 2020 datasets, developed for the
Knowledge Discovery in Healthcare Data Blood Glucose Level Predication Challenge (Marling &
Bunescu, 2018; 2020).
The data pertain to 12 individuals, each with approximately 10,000 5-minutesamples for training and 2,500 for testing.
12% of glucose values are missing, but we do not includewindows with missing glucose values.
We do not include windows with more than one carbohydratemeasurement in our analysis. We sum carbohydrates to the first timepoint if they are less than 15minutes apart to maximize the amount of usable data.
We include only individuals with at least 100training carbohydrate measurements, as fewer than this are not sufficient for learning a model.
6. B ASELINES AND U PPER B OUND
For all non-coteaching methods, we train two DAEs in parallel and report results on their averagedoutput for a fair comparison.
We also note that all models receive the same auxiliary variables (bloodglucose/ insulin) as input in an identical fashion.
• CAE :
An upper performance bound.
This model is an autoencoder trained with clean data, whichwe would expect to perform better than any method without access to clean data.
• NAC : Our first baseline.
A DAE that treats the noisy data as clean which has been shown toperform well in low noise settings (Xu et al., 2020).
• NR2N : Our second baseline is noisier2noise (Moran et al., 2019), which uses the known noisedistribution to recover the clean signal.
NR2N trains similarly to NAC , but at evaluation time atransform is used to recover the clean values (briefly, if the distribution of N is known and we setM = N , the model should learn to recover half of the noise so the value used at evaluation is 2ˆ y − z ).•
SUP : Our motivating setting can be re-framed as a supervised learning problem: predict y (or x )values using b values as input.
Depending on the noise distribution, it is possible that a model trainedon noisy y values could learn to predict the correct x , using similar logic to that found in Lehtinenet al.
We therefore use this supervised setting as a naive baseline.
We simply input b to thesame network used in the DAE setting and calculate loss as (ˆ y − y ) 2 during training, but here themodel has no information regarding y or z .
As in the DAE setting, at test time we evaluate (˜ y − x ) 2 .•
SUPCT : We apply co-teaching to the supervised setting ( SUP ), to ensure that performance gainsobserved are due to the combination of DAEs and co-teaching, and not co-teaching alone.
Here, themodel is tuned and trained identically to N + 2N , except the model does not receive y or z values.
6. I MPLEMENTATION & T RAINING D ETAILS
Each DAE is implemented as a 2-layer bidirectional LSTM with 100 hidden units.
The final hiddenstate is passed to a FC layer with a single output.
The output of the model is added to the inputvalue corresponding to y , so that the network is tasked with learning the error term rather than acomplete reconstruction.
This initializes the network’s output to be the identity (provided the outputis initially low magnitude), which is useful for addressing biased data.
Because we only aim tocorrect a single carbohydrate ( y ) value but use a time-based model, we set one dimension of themodel input to be y for all timepoints.
Blood glucose values b , bolus and basal insulin are inputto the model as time-series.
We also carry over bolus insulin values (which occur sparsely) to theend of the input window, to increase their impact on gradient calculations.
We threshold the outputof each DAE at 0, because carbohydrates (our x and y ) values cannot be negative.
For each coteaching method and NR2N , we tune on a single subject’s validation data (adult#001) and use thefound parameters for all further analyses.
Tuning is described in Appendix A .
At evaluation wereport the result of the average correction learned by both networks when y values are given as input( i.e. , where ˜ y
i
= DAE i ( y, b ) , we report L ( x, (˜ y
+ ˜ y 2 ) / 2) ).
We split each dataset into training, validation and test sets used for evaluation purposes.
For thesimulated dataset, we use 80 days for training, 20 for validation, and 50 for testing.
For the realdataset, we split the training data into 80% train and 20% validation.
The held-out test data wereused for evaluation only.
We implement and train our models in Pytorch 1.9.1 with CUDA version10.2, using Ubuntu 16.04.7, a GeForce RTX 2080, an Adam optimizer (Kingma & Ba, 2014), and abatch size of 500.
We use a learning rate of 0.01 and a weight decay of 10 − 7 .
We train for at least500 iterations, and then until validation performance does not improve for 50 iterations, selectingthe model for which validation performance was best.
For both datasets, we train and test a model oneach individual and report across-individual averages.
Such individual-specific models/evaluationsare common in blood glucose control and forecasting (Silvia Oviedo, 2016), since dynamics varygreatly across individuals and individual-specific training data are typically available
We perform co-teaching on samples containing non-zero y values only.
However, when training allmodels (including baselines) we also pass zero-valued y samples (and their corresponding b values)through both DAEs and take loss equal to ˆ y 2 for these samples.
We do this because there are manymore samples with zero-valued carbohydrates than there are with positive values, and this allowsthe models to learn from this larger corpus.
We report results on only positive-valued y s, becausedenoising is only applied to such values.
For sample selection during co-teaching, we use mean squared percentage error ( 100% · ((ˆ y − y ) /y ) 2 )to avoid eliminating all high-valued y samples, as they are likely to have higher noise values.
As anoise function during training, we use z = (1 + N (0 , . 5))
Bern ( . 5)
y , i.e. we add random noise tohalf of the samples so that the model can learn to utilize noisy z information, and zero-out the otherhalf so that the model has to learn to distinguish zero from non-zero y values based on b alone.
6. E VALUATION
Simulated Data Metrics.
When clean carbohydrate values are available, we use MSE between thecleaned carbs and true carbohydrate values as our metric to report the remaining noise ( mean (( x −˜ y ) 2 ) , where ˜ y
=
(˜ y
+ ˜ y 2 ) / 2 ).
The lower this value, the more noise has been removed.
Althoughwe assume that these data are not available at training time, we use them for evaluation.
Since it canbe difficult to interpret the meaning of a difference in MSE, we also consider a clinically motivatedevaluation metric: time in range .
Time in range is a measure of blood glucose management andvaries with the accuracy of the carbohydrate measurements.
The more accurate the carbohydrateestimates the more time an individual will spend ‘in range.’
Here, we run a simulation of the subjectof interest with the default basal-bolus controller using bolus values calculated from the updatedcarbohydrate values, and report the proportion of time in the simulation that each individual spentwith blood glucose values between 70 and 180, or the euglycemic/healthy range.
This metric servesto indicate the real-world impact each approach might have.
For both metrics, 95% confidenceintervals are calculated for each subject using 1,000 bootstrap re-samples, and the average 2.5 and97.5 percentiles across subjects are reported.
Sensitivity analyses.
To evaluate our model under different noise assumptions, we repeat our analysis with multiple noise generation methods ( x → y ), without altering hyperparameters or our y → z noise function.
We use various Gaussian and uniform distributions reported in Appendix B , whichinclude zero and negative mean multiplicative and additive noise functions.
We do not aim at a comprehensive evaluation of all possible noise types, but rather we aim to include various distributionsthat are likely similar to those that might arise in our motivating domain.
Real Data analysis.
Without access to clean carbohydrate values at test time for the real dataset(unlike the simulated dataset), we evaluate the performance of our denoising approach based ona proxy.
We take advantage of the fact that poorly estimated carbohydrates result in inaccuratebolus calculations, which result in poor blood glucose management.
We expect that inaccuratecarbohydrates estimates (large [ x − y ] values) result in the poor blood glucose management.
For amodel that has come close to estimating x correctly, we would observe a correlation between [ ˜ y − y ]values and blood glucose control in the time period following a meal.
We assess this with CorrectionRisk-Correlation, henceforth referred to as CRC, defined as the Spearman correlation between thesquared carbohydrate correction value ( (˜
y − y
and the average Magni Risk (Magni et al., 2007)of blood glucose in the second hour following the carbohydrate.
We use the Spearman correlation toaccount for non-linearities in the risk and correction value distributions.
Magni risk is a measure ofhow far from a safe value blood glucose is; higher risk values correspond to blood glucose values thatare either dangerously high or dangerously low.
We use the second hour following the carbohydratebecause the effects of the carbohydrate consumption and insulin bolus have not fully taken effect inthe first hour.
We calculate this correlation across all carbohydrates observed in all individuals.
Forvalidation purposes, we also calculate this metric for the simulated dataset.
R ESULTS AND D ISCUSSION
Through our experiments, we aim to answer the following questions.
1) Does our approach meaningfully reduce error across a variety of simulated individuals, compared to existing approaches? 2)
Is our model robust to different domain-appropriate noise distributions?
3) Does our model showstrong performance in real data, indicating accurate denoising?
[Table] [Table]
[Table]
0.22, 9eError Reduction for simulated data.
Co-taught denoising autoencoders, N + 2N , outperform baselines in terms of noise reduction (MSE) ( Table 1)
N + 2N reduces MSE to 18 g 2 , but fall short ofthe value achieved by the clean-data DAE (7 g 2 ), as expected.
CAE does not achieve perfect MSE,probably due to insufficient training data or to a small amount of noise in the CGM signal.
Ourapproach’s reduction in noise is meaningful since it leads to significantly better time in range.
Mostmethods offer an improvement in % time in range when used in a basal bolus controller, with baselines increasing over the noisy value from 65% to 69%, and N + 2N further improving performanceto 71%, recovering 6% time in range out of a total of 8% lost when using noisy versus clean values.
SUPCT performs worse than any other method including SUP , likely because without the noisy carbohydrate measurement as input, co-teaching cannot learn the relationship between b and y as easily,and therefore does not identify the less corrupted samples during training.
This results in essentiallyrandom sub-sample selection, hampering performance as less training data becomes available.
SUPdoes not suffer from this problem because it utilizes the entire dataset for all iterations.
Alternate noise assumptions.
N + 2N outperforms all baselines across the majority of noise distributions ( Figure 2).
For zero-mean uniform multiplicative noise, NAC outperforms the proposed approach.
We hypothesize that NAC performs well in this setting because the expected value of thenoise is zero and the variance is lower than in other settings (it is 33%, which is approximately 30g,compared to 75% in the multiplicative normal setting, or 40g and 60g in the additive noise settings,see Appendix B ), which is the setting for which NAC has been shown to perform best.
Of note, thisanalysis was carried out without additional tuning, demonstrating the resilience of our approach tovarying noise assumptions.
Across biased noised distributions, our proposed approach consistentlyoutperforms all baselines.
This resilience is likely due to our approach’s identity-initialized sampleselection and a lack of dependence on matching M and N noise distributions.
Experiments on Real Data.
For the real dataset, N + 2N outperforms all baselines with respectto CRC.
For simulated data, we see that, without exception, models with lower remaining MSEafter denoising have a higher or equal CRC.
This indicates that our metric serves as a reasonableproxy for remaining error when true values are unavailable.
Plots showing the components used tocalculate CRC (magnitude of carbohydrate correction versus Magni risk an hour after the meal) canbe found in Appendix C .
Interestingly, the SUP baseline performs fairly well for this task on the realdataset.
We hypothesize that this may be because carbohydrate measurements are so unreliable forthis dataset that learning to predict them from scratch (without access to noisy values at test time) issufficient.
Four out of twelve individuals in the real dataset had too few carbohydrate measurementsto be included in our analysis, indicating that more work would need to be done for our algorithm tobe applicable to a broader population.
C ONCLUSIONS
We propose a new approach to denoising, ‘Noise + 2Noise’, that does not assume access to cleansamples and applies to time-series data.
Our approach leverages an auxiliary time-series that is related to the target signal to help identify target samples with less noise.
Our approach is the first toadapt co-teaching to de-noising.
In the context of carbohydrate recovery for blood glucose management, compared to existing approaches, ‘Noise + 2Noise’ leads to better signal reconstruction that isboth statistically significant and clinically significant.
While promising, our approach is not withoutlimitations.
Our primary analyses are on simulated data where ground truth labels are available, butin real datasets common evaluation metrics (e.g., MSE) do not apply and we must rely on proxies.
Aspresented, our approach is designed for retrospective carbohydrate correction; more work is necessary to investigate its applicability to closer-to-real-time correction.
While our approach shows clearsignificance for the individuals available, evaluation on a much larger population would be necessaryprior to widespread adoption.
Finally, while we have empirically shown that co-teaching appears toselect a low-noise sample, we have not proven any statistical guarantees.
Despite these limitations,we have demonstrated that is is feasible to correct a noisy variable without access to clean samples,expanding the utility of ideas the image analysis and noisy label learning to time-series reconstruction.
Applied to domains in which datastreams are composed of both individual-reported data anddata measured from relioable sensors (e.g., mHealth), or datastreams composed of series with disparate noise levels (e.g., external vs internal temperature, multiple audio recordings), our approachcould aid in improving the reliability of uncertain time-series data.
A PPENDICES ( SUPPLEMENTAL )A T UNING DETAILS
For each model, tuning was performed on simulated adult#001 using validation performance.
No additional tuning was performed for other individuals or noise functions.
For Noisier2Noise, we selected α , the parameter that controls the relative noise distributions, from[0.1,0.3,0.5,0.7,0.9,1.0,1.25,1.5,1.75,2], ultimately selecting α = 1 .
Because we do not assumeaccess to the exact noise we would not expect this method to perform spectacularly, but note that itoften outperforms other baselines.
For co-teaching methods, we performed a simple grid search over the values of E k =[250,500](where 500 is the minimum number of training iterations), τ =[0.333,0.5,0.667], and σ =[0.1,0.3,0.5,0.7].
For N + 2N , we selected E k = 250 , τ = 0 .
333 , and σ = 0 .
For the supervised setting co-teaching ( SUPCT ), we set selected T k = 500 , τ = 0 .
5 and σ = 0 .
B A LTERNATE NOISE FUNCTIONS
We consider noise functions that might arise in carbohydrate counting.
None are highly dissimilarfrom our main analysis noise function: we aim here at feasibility, rather than a comprehensive surveyon a broad selection of loss functions, which our method would likely be unable to address withoutfurther tuning or modification.
Here, U ( a, b ) denotes a uniform distribution with values between aand b .
Carbohydrate values range between 0 and 200.
After adding noise, y values are capped aboveand below by 1 and 200.
Alternate noise functions include:
Zero-mean multiplicative Gaussian: y = (1 + N (0 , .
x2.
Negative-mean multiplicative Gaussian (primary noise function):
y = (1 + N ( − . 25 , . 5))
x3.
Zero-mean additive Gaussian: y
= x + N (0 , 40)4.
Negative-mean additive Gaussian: y
= x + N ( − 30 , 50)5.
Zero-mean multiplicative Uniform: y
= U ( . 5 , 1 . 5) x6.
Negative-mean multiplicative Uniform: y
= U (0 , 1 . 6) x7.
Zero-mean additive Uniform: y
= x + U ( − 60 , 60)8.
Negative-mean additive Uniform: y
= x + U ( − 60 , 40)
C CRC P LOTS
With N + 2N , we see a higher correlation between the magnitude of carbohydrate correction and riskfollowing the meal compared to baselines for both real and simulated data ( Figure 3 ).