Paper:
VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION
We study the problem of learning domain invariant representations for time series data while transferring the complex temporal latent dependencies between domains. Our model termed as Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) is built atop a variational recurrent neural network (VRNN) and trains adversarially to capture complex temporal relationships that are domain-invariant. This is (as far as we know) the first to capture and transfer temporal latent dependencies of multivariate time-series data. Through experiments on real-world multivariate healthcare time-series datasets, we empirically demonstrate that learning temporal dependencies helps our model’s ability to create domain-invariant representations, allowing our model to outperform current state-of-the-art deep domain adaptation approaches.
1 INTRODUCTION
Many real-world applications require effective machine learning algorithms that can learn invariant representations across related time-series datasets. For example, precision medicine for patients of various age groups, mobile application recommendation for users based on locations, and so on. In these examples, while the domains (i.e. age group and location) may vary, there exist common predictive patterns that can aid in inferring knowledge from one domain to another. More often than not, some domains have a significantly larger number of observations than others (e.g., respiratory failure in adults vs. children). Therefore effective domain adaption of time-series data is in great demand.
The general approach to tackling domain adaptation has been explored under many facets which include reducing the domain discrepancy between the source and target domains(Ben-David et al. (2007)), instance re-weighting (Jiang & Zhai (2007)), subspace alignment (Fernando et al. (2013)), and deep learning (Tzeng et al. (2015); Ganin & Lempitsky (2014)). Many of these approaches work very well for non-sequential data but are not suitable for multivariate time-series data as they do not usually capture the temporal dependencies present in the data. For sequential data, earlier work has successfully used dynamic Bayesian Networks(Huang & Yates (2009)) and Recurrent Neural Networks (Socher et al. (2011)) to learn latent feature representations which were domaininvariant. Unfortunately, these works were not flexible enough to model non-linear dynamics or did not explicitly capture and transfer the complex latent dependencies needed to perform domain adaptation of time-series data.
In this paper, we address this problem with a model that learns temporal latent dependencies (i.e. dependencies between the latent variables across timesteps) that can be transferred across domains that experience different distributions in their features. We draw inspiration from the Variational Recurrent Neural Network (Chung et al. (2016)) and use variational methods to produce a latent representation that captures underlying temporal latent dependencies. Motivated by the theory of domain adaptation (Ben-David et al. (2010)), we perform adversarial training on this representation
*: Co-first authors
t-SNE projections for the latent representations of DNN, R-DANN, and our VRADA model. We show adaption from Adult-AHRF to Child-AHRF data. Source data is represented with red circles and target data with blue circles. From left to right, one can see that domain adaptation results in mixing the source and target domain
data distributions. We can also see a story of how encoding more temporal dependency into the latent representation induces more domain-invariant representations. As models capture more underlying factors of
variation, post domain adaptation representations gradually smoothen and become evenly dispersed, indicating that temporal dependency acts synergestically with domain adaptation.
similarly to the Domain Adversarial Neural Network (DANN) (Ganin et al. (2016)) to make the representations invariant across domains. We call our model the Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model. As far as we know, this is the first model capable of accomplishing unsupervised domain adaptation while transferring temporal latent dependencies for complex multivariate time-series data. Figure 1 shows an example of the domain invariant representations learned by different deep learning models including our VRADA model. From this figure, we can see that our model (VRADA) shows better mixing of the domain distributions than the competing models indicating that it learns better domain invariant representations.
In order to prove the efficacy of our model, we perform domain adaptation using real-world healthcare time-series data. We choose healthcare data for two primary reasons. (1) Currently, a standard protocol in healthcare is to build, evaluate, and deploy machine learning models for particular datasets that may perform poorly on unseen datasets with different distributions. For example, models built around patient data from particular age groups perform poorly on other age groups because the features used to train the models have different distributions across the groups (Alemayehu & Warner (2004); Lao et al. (2004); Seshamani & Gray (2004)). Knowledge learned from one group is not transferrable to the other group. Domain adaptation seems like a natural solution to this problem as knowledge needs to be transferred across domains which share features that exhibit different distributions. (2) Healthcare data has multiple attributes recorded per patient visit, and it is longitudinal and episodic in nature. Thus, healthcare data is a suitable platform on which to study a model which seeks to capture complex temporal representations and transfer this knowledge across domains.
The rest of the paper is structured as follows. In the following section, we briefly discuss the current state-of-the-art deep domain adaptation approaches. Afterwards, we present our model mathematically, detailing how it simultaneously learns to capture temporal latent dependencies and create domain-invariant representations. In Section 4, we compare and contrast the performance of proposed approach with other approaches on two real-world health care datasets, and provide analysis on our domain-invariant representations.
2 RELATED WORK
Domain adaptation is a specific instance of transfer learning in which the feature spaces are shared but their marginal distributions are different. A good survey on the two has been done in several previous works (Pan & Yang (2009); Jiang (2008); Patel et al. (2015)). Domain adaptation has been thoroughly studied in computer vision(Saenko et al. (2010); Gong et al. (2012); Fernando et al. (2013)) and natural language processing (NLP) (Blitzer (2007); Foster et al. (2010)) applications. Recently, the deep learning paradigm has become popular in domain adaptation (Chen et al. (2012); Tzeng et al. (2015); Yang & Eisenstein; Long & Wang (2015)) due to its ability to learn rich, flexible, non-linear domain-invariant representations. Here, we briefly discuss two deep domain adaptation approaches which are closely related to our proposed model. Domain Adversarial Neural Networks (DANN)
(Ganin et al. (2016)) is a deep domain adaptation model which uses two core components to create domain-invariant representations, a feature extractor that produces the data’s latent representation, and an adversarial domain labeler that attempts to classify that data’s domain to help the feature extractor produce latent representations which are domain-invariant. In Louizos et al. (2015), the authors propose Variational Fair AutoEncoder, which uses Variational Autoencoding architecture (Kingma & Welling (2013)) to learn latent representations where most of the information about certain known factors of variation are purged from the representation while still retaining as much information about the data as possible. While, these deep learning approaches learn domain-invariant representations, they fail to capture and transfer the underlying complex temporal latent relationships from one domain to another as they use convolutional or feed forward neural networks which we claim are not suitable for multivariate time-series data.
Other works such as Huang & Yates (2009); Xiao & Guo (2013) have used distributed representations for domain adaptation in NLP sequence labeling tasks. However, they either induce hidden states as latent features using dynamic Bayesian networks (DBNs) or learn generalizable distributed representations of words using Recurrent Neural Networks (RNN) (Socher et al. (2011)) to enable domain adaptation. These works either model the highly non-linear dynamics, as one can with RNN, or capture the complex latent dependencies present in sequential data, as one can with DBNs, but not both. To overcome the challenges of DBNs and RNNs, Variational Recurrent Neural Network (VRNN)( Chung et al. (2016)) was proposed recently to capture the complex relationship between the underlying hidden factors of variation and the output variables at different time-steps. The VRNN uses Variational Autoencoders (VAEs)( Kingma & Welling (2013); Goodfellow et al. (2016)) at each time-step to learn a complex relationship between the latent hidden factors across time-steps. Like the VAE, its latent variable is parametric. Combined, these things make it well-suited for multimodal sequential data such as multivariate time-series. In the following section, we discuss our approach, Variational Adversarial Deep Domain Adaptation (VRADA), which uses a VRNN to model and transfer complex domain-invariant temporal latent relationships for unsupervised domain adaptation of multivariate time-series.
3 VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION
In this section, we present our Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model for the purpose of capturing and transferring temporal latent dependencies across domains via domain-invariant representations. First, we introduce the notations used in this paper and then discuss our VRADA model in detail.
3.1 NOTATIONS
Let us denote a multivariate variable-length time series with N data samples as {xi = (xit)T i t=1}Ni=1, where xit ∈ RD. (Note: in our experiments, for all data samples T i = τ , but for generality we maintain T i). We denote {xiS}ni=1 as source domain data and {xiT }Ni=n+1 as target domain data. We assume that each source domain data sample xiS comes with L labels yi ∈ {0, 1}L (for example, these labels may correspond to a clinical outcome such as mortality or ICD9 diagnosis codes), while
target domain has no labeled data samples. We assign a domain label di ∈ {0, 1} to each data sample to indicate if it comes from the source or target domain. di will be used for adversarial training.
3.2 VRADA
The block diagram of our VRADA model is shown in Figure 2. To explicitly model the dependencies between the latent random variable across time steps, the VRADA model utilizes Variational Recurrent Neural Networks (VRNN) (Chung et al. (2016)). The VRNN effectively contains a Variational AutoEncoders (Kingma & Welling (2013)) at every time step, all of which are conditioned on previous auto-encoders via the hidden state ht−1 of an RNN, such as an LSTM (Hochreiter & Schmidhuber (1997)). Therefore, for each time-step of xit, we infer a latent random variable z i t via
zit|xit ∼ N (µz,t, diag(σz,t)), where [µz,t, σz,t] = ϕencτ (ϕxτ (xit), ht−1)
with prior zit ∼ N (µ0,t, diag(σ0,t)), where [µ0,t, σ0,t] = ϕpriorτ (ht−1) where µ∗,t, σ∗,t denote parameters of a generating distribution, and ϕ∗τ can be any highly flexible function such as deep neural networks. For each zit, x i t is generated via
xit|zit ∼ N (µx,t, diag(σx,t)), where [µx,t, σx,t] = ϕdecτ (ϕzτ (zit), ht−1)
and learned by optimizing the VRNN objective function: Lr(xit; θe, θg) = Eqθe (zi≤Ti |xi≤Ti )[ T i∑ t=1 (−D(qθe(zit|xi≤t, zi<t)||p(zit|xi<t, zi<t))+log pθg (xit|zi≤t, xi<t))])
where qθe(z i t|xi≤t, zi<t) is the inference model, p(zit|xi<t, zi<t) is the prior, pθg (xit|zi≤t, xi<t) is the generative model, θe is the parameters of the VRNN’s encoder, θg the parameters of the VRNN’s decoder, and D(·||·) refers to KL-Divergence. Note: z≤T refers to the set of all zt such that t ≤ T , likewise for z<T . For each xi, we use z̃i ∼ qθe(ziT i |x i ≤T i , z i <T i) as our feature representation for source domain classification task since it captures temporal latent dependencies across the time-steps. Training the VRNN for the source domain classification involves solving the following optimization:
min θe,θg,θy
1
n n∑ i=1 1 T i Lr(xi; θe, θg) + 1 n n∑ i=1 Ly(xi; θy, θe) + λR(θe) (1)
whereR(θe) is a regularizer for the parameters of VRNN encoder (which is also the feature extractor of VRADA) with a tuning hyperparameter λ.
As we are interested in achieving domain adaptation via the latent representation z̃i (i.e. to make z̃i domain-invariant), we can adversarially train the above objective function (equation 1) by employing the domain adaptation idea proposed in Ganin et al. (2016). Let Gy(z̃i; θy) and Gd(z̃i; θd) represent the source label classifier (to predict source labels yi) and domain label classifier (to predict domain labels di) respectively with parameters θy and θd for a given input z̃i. Here, Gy(.) and Gd(.) can be deep neural networks. Let us denote their loss functions respectively as
Ly(xi; θy, θe) = LB(Gy(Ve(xi; θe); θy), yi); Ld(xi; θd, θe) = LB(Gd(Ve(xi; θe); θd), di)
where LB is the classification loss such as a binary or categorical cross-entropy loss function and Ve(x i; θe) is the VRNN encoder that maps input xi to z̃i.
Now, for adversarial training, we consider the following domain adaptation term as the regularizer of equation 1.
R(θe) = max θd [ − 1 n n∑ i=1 Ld(xi; θd, θe)− 1 n′ N∑ i=n+1 Ld(xi; θd, θe) ]
(2)
where n′ is the number of target domain samples. As shown in Ganin et al. (2016),R is the domain regularizer and it is derived from the empiricalH−divergence between the source domain and target domain samples( Ben-David et al. (2010)).
Combining the joint optimization problem of equations 1 and 2 leads to our VRADA model, where we minimize the source classification risk and at the same time achieve domain adaptation. Mathematically, we optimize the following complete objective function:
E(θe, θg, θy, θd) = 1
N N∑ i=1 1 T i Lr(xi; θe, θg)+ 1 n n∑ i=1 Ly(xi; θy)−λ( 1 n n∑ i=1 Ld(xi; θd)+ 1 n′ N∑ i=n+1 Ld(xi; θd)))
(3)
where λ is a trade-off between optimizing on making domain-invariant representations and optimizing source classification accuracy. Our optimization involves minimization with respect to some parameters, and maximization with respect to the others, i.e., we iteratively solve the following:
(θ̂g, θ̂y, θ̂e) = arg min θg,θy,θe E(θe, θg, θy, θ̂d)
θ̂d = argmax θd E(θ̂e, θ̂g, θ̂y, θd)
with the gradient updates calculated as:
θe ← θe − η(∂Lr∂θe + ∂Ly ∂θy − λ∂Ld∂θd ) (4)
θg ← θg − η ∂Lr∂θg (5) θd ← θd − η ∂Ld∂θd (6) θy ← θy − ηλ∂Ly∂θy (7)
where η is the learning rate. We can use stochastic gradient descent (SGD) to solve the equations (5-7). To solve equation (4), we can use SGD and the gradient reversal layer (GRL)(Ganin et al. (2016)). The role of GRL is to reverse the gradient sign while performing backpropagation. This ensures that the domain classification loss is maximized which makes the feature representations domain-invariant.
Thus, VRADA results in learning feature representations which are domain-invariant (due to domain regressorR) and which capture the temporal latent dependencies (due to optimizing VRNN objective function Lr). These things combine to allow the VRADAs’ discriminative power on the source domain to transfer to the target domain.
4 EXPERIMENTS
We conduct experiments on two real-world health care datasets to answer the following questions: (a) How does our VRADA model perform when compared to the state-of-the-art domain adaptation and non-adaptation approaches? (b) How different are the domain-invariant representations learned by various domain adaptation methods? (c) How do we show that the temporal latent dependencies are transferred between domains? In the remainder of this section, we will describe the datasets, methods, empirical results, and show visualizations to answer the above questions.
4.1 DATASET DESCRIPTION
We conduct experiments on two health care datasets, including the MIMIC-III dataset and a Pediatric ICU (PICU) dataset from Children’s Hospital Los Angeles.
MIMIC-III( Johnson et al. (2016)) is a public dataset with deidentified clinical care data collected at Beth Israel Deaconess Medical Center from 2001 to 2012. It contains over 58,000 hospital admission records of 38,645 adults and 7,875 neonates. For our experiments, we extracted the following two datasets:
• Adult-AHRF dataset: To study domain adaptation for adult patients with acute hypoxemic respiratory failure (AHRF), we extracted 20 time series features (such as Base excess, blood pH value, Mean Air Pressure, PaO2, etc.) from 5527 admission records based on Khemani
et al. (2009). We grouped the patients into 4 groups/cohorts based on their age[1] - Group 2: working-age adult (20 to 45 yrs, 508 patients); Group 3: old working-age adult (46 to 65 yrs, 1888 patients); Group 4: elderly (66 to 85 yrs, 2394 patients); Group 5: old elderly (85 yrs and up, 437 patients). We treated each group as a separate domain with which we could perform domain adaptation. For each patient, we used the first 4 day after admission (with each day serving as a single time-step) as time series data for training and testing our models. • ICD9 dataset: For this dataset we extracted 99 time series features from 19714 admission
records from 4 modalities including input-events (fluids into patient, e.g., insulin), outputevents (fluids out of the patient, e.g., urine), lab-events (lab test results, e.g., blood pH values, platelet count, etc.) and prescription-events (drugs prescribed by doctors, e.g., aspirin, potassium chloride, etc.). These modalities are known to be extremely useful for monitoring ICU patients. All the time series are of more than 48 hours of duration, and only the first 24 hours (after admission) 2-hourly sampled time series data is used for training and testing our models. We use this dataset to predict the ICD9 Diagnosis code categories for each patient’s admission record.
Child-AHRF dataset: This is a PICU dataset which contains health records of 398 children patient with acute hypoxemic respiratory failure in the intensive care unit at Children’s Hospital Los Angeles (CHLA)(Khemani et al. (2009)). Similar to Adult-AHRF, this dataset has 20 time series features collected for 4 days after ICU admission. This dataset is considered as one group (Group 1: children, age 0 to 19 yrs) and represents one domain.
4.1.1 PREDICTION AND DOMAIN ADAPTATION TASKS
Mortality Prediction: For Adult-AHRF and Child-AHRF datasets, we are interested in predicting mortality, i.e. whether a patient dies from AHRF during their hospital stay. 20.10% of all the patients in Child-AHRF and 13.84% of all patients in Adult-AHRF have a positive mortality label (i.e. the patients who die in hospital).
ICD9 Code Prediction: Each admission record in MIMIC-III dataset has multiple ICD-9 diagnosis codes. We group all the occurrences of the ICD-9 codes into 20 diagnosis groups[2] . For the ICD9 dataset, we are interested in predicting these 20 ICD-9 Diagnosis Categories for each admission record. We treat this as a multi-task prediction problem.
Domain Adaptation Tasks: We study unsupervised domain adaptation (i.e. target domain labels are unavailable during training and validation) task with-in age groups of Adult-AHRF dataset, ICD9 dataset and across Adult and Child-AHRF datasets. For Adult-AHRF and ICD9 datasets, we created 12 source-target domain pairs using the age groups, pairing up each domain Di with another domain Dj 6=i, for example, the source-target pair 2-5 was used for adapting from group 2 (working-age adult) to group 5 (old elderly). We also created 4 source-target pairs for performing domain adaptation from 4 adult age-groups to 1 child age-group.
4.2 METHODS AND IMPLEMENTATION DETAILS
We categorize the methods used in our main experiments into the following groups:
• Non-adaptive baseline methods: Logistic Regression (LR), Adaboost with decision regressors (Adaboost), and feed forward deep neural networks (DNN)
• Deep Domain adaptation methods: Domain Adversarial Neural Networks (DANN) (Ganin et al. (2016)); DANN with a RNN (LSTM) as feature extractor (R-DANN); Variational Fair Autocoder (VFAE)(Louizos et al. (2015))
• Our method: Variational Recurrent Adversarial Deep Domain Adaptation (VRADA)[3] .
[1]:https://www.cms.gov/Research-Statistics-Data-and-Systems/ Statistics-Trends-and-Reports/NationalHealthExpendData/
[2]: http://tdrdata.com/ipd/ipd_SearchForICD9CodesAndDescriptions.aspx. “Conditions Originating in the Perinatal Period” is not present in the preprocessed dataset.
[3]: Codes will be publicly released soon
In all our experiments, we conducted unsupervised domain adaptation where target domain labels are unavailable during training and validation. For R-DANN, we used LSTM(Hochreiter & Schmidhuber (1997)) as the feature extractor network instead of the feed-forward neural networks used in DANN. For VFAE, DANN and all the non-domain adaptive approaches we flattened the time series along time axis and treat it as the input to the model. For fairness, the classifier and feature extractors of the VRADA and R-DANN were equivalent in depth and both had the same model capacity. We also ensure that the size of latent feature representation z̃i are similar for VRADA and DANN models. The model capacity of VFAE was chosen to be similar to VRADA. All the deep domain adaptation models including ours had depth of size 8 (including output classifier layers). We used the Adam optimizer ( Kingma & Ba (2014)) and ran all models for 500 epochs with a learning rate of 3e−4. We set an early stopping criteria that the model does not experience a decrease in the validation loss for 20 epochs. Source domain data was split into train/validation subsets with a 70/30 ratio and target domain data into train/validation/test subsets with a 70/15/15 ratio. In order to compare all the methods, we report AUC scores on the entire target domain set, and the test subset for each target domain data of a source-target pair.
4.3 QUANTITATIVE RESULTS
In Table 1, we compare non domain adaptation and domain adaptation models’ performance on the target domain test subset for the AHRF mortality prediction task. It is immediately clear that domain adaptation methods consistently outperform non domain adaptation methods. We see that generally the VRADA outperforms both variants of the DANN with it consistently seeing scores ∼ 4% higher. While the standard deviation for the VRADA was about 1%, it was about 2% for the R-DANN, further showing our models efficacy as it converges to more stable local optima. Our model VRADA beats state-of-the-art DANN(Ganin et al. (2016)) and VFAE(Louizos et al. (2015)) on all the source-pair domain adaptation tasks for Adult-AHRF dataset. For the domain adaptation from Adult-AHRF to Child-AHRF dataset, we observe that VRADA mostly outperforms all the competing models. This shows that our model can perform well even for smaller target domain datasets.
In the above table, we test classification without adaptation using Logistic Regression (LR), Adaboost with decision tree classifiers and Feed forward Deep Neural Networks (DNN); and with adaptation using Deep Domain Adversarial Neural Networks (DANN), a DANN with an LSTM in its feature extractor (R-DANN), Variational Fair Autoencoder (VFAE) and our Variational Adversarial Domain Adaptation Model (VRADA). All results are reported on the target domain test subset dataset.
As the AHRF mortality prediction task made it clear that domain adaptation is necessary for intergroup adaptation, for the ICD9 multi-task prediction task that involved data with time-steps of length 12, we focused strictly on domain adaptive models (i.e. the DANN, R-DANN, and VRADA). Table 2 shows the aggregated AUC scores on the entire target domain dataset and test data of the target domain for the 20 tasks of the ICD9 Code Prediction task. Here, we clearly see that VRADA and
Here, we compare results for the ICD9 Diagnosis Code Prediction task on the ICD9 dataset. For each model, the top row corresponds to the performance on the entire target domain dataset and the bottom row corresponds to performance on the test subset (15%) of the target domain dataset.
R-DANN models outperform DANN Ganin et al. (2016) by significant margins. We also observe that VRADA outperforms R-DANN by 1.5 ∼ 2% when averaged over all the source-target domain pairs.
4.4 DISCUSSION
Figure 3 shows the temporal latent dependencies captured by our VRADA as compared to the R-DANN for 3-4 source-target pair. While both models learn temporal latent dependencies fairly well, the VRADA outperforms the R-DANN in two ways. First, the VRADA’s neurons learned stronger predictions of whether features are relevant towards modeling the data. If we look at the VRADA row, for both AHRF and ICD9 we see that the neural activation patterns are more consistent across time-steps than for R-DANN. Figure 4 shows the unrolled memory cell states (in the form Examples× (Time ∗ Neurons)) for all the source and target domain data points. We see a consistent activation firing patterns across all these data points for VRADA but not for R-DANN. Together with the stronger performance on 3-4 for AHRF and 2-5 for ICD9, this potentially indicates that VRADA is better learning the temporal dependencies.
Second, nuanced values are consistent across time-steps for the VRADA, exhibiting a gradual transition towards stronger activation with time, whereas the temporal activation pattern of the RDANN seems somewhat sporadic. While activation gradients across time are consistent for both the R-DANN and VRADA, more consistent inhibitory and excitatory neuron firing patterns indicate that the VRADA better transfers knowledge. Another indication of domain adaptation was shown in Figure 1c. Looking at the t-SNE projections of feature representations of DNN, R-DANN, and VRADA we can see that the addition of temporal latent dependencies might help in better mixing of the domain distributions since we observe that the data is more evenly spread out. Figure 1c and Figure 3 together indicate that the VRADA’s temporal latent dependency capturing power and ability to create domain-invariant representations act synergistically. For plots of activation patterns without domain adaptation, please see appendix section 6.2.3.
5 SUMMARY
Because of its diverse range of patients and its episodic and longitudal nature, healthcare data provides a good platform to test domain adaptation techniques for temporal data. With it as our example, we showcase the Variational Recurrent Adversarial Domain Adaptation (VRADA) model’s ability to learn temporal latent representations that are domain-invariant. By comparing our model’s latent representations to others’, we show its ability to use variational methods to capture hidden factors of variation and produce more robust domain-invariant representations. We hope this work serves as a bedrock for future work capturing and adapting temporal latent representations across domains.
ACKNOWLEDGMENTS
This material is based upon work supported by the NSF research grants IIS-1134990, IIS-1254206, Samsung GRO Grant and the NSF Graduate Research Fellowship Program under Grant No. DGE1418060. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies. We also acknowledge Thailand’s Development and Promotion of Science and Technology Talents Project for financial support. We thank Dr. Robinder Khemani for sharing the Child-AHRF dataset.
6 APPENDIX
6.1 TRAINING VARIATIONS
We tested 3 variations of training VRADA: (a) training VRADA regularly as discussed in Section 3 (denoted by I), (b) loading a pretrained VRNN encoder and optimizing strictly off the classification errors, i.e.
E(θe, θy, θd) = 1
n n∑ i=1 Ly(xi; θy)− λ( 1 n n∑ i=1 Ld(xi; θd) + 1 n′ N∑ i=n+1 Ld(xi; θd))) (8)
and (c) loading a pretrained VRNN encoder and using the objective as presented in equation 3 (denoted by III). Key to note is that in method II, we do not apply variational methods towards learning the shared latent representation. This was done to test whether they were helpful or harmful towards the learned latent representation used for classification. In method III, we train VRADA as normal but load a pretrained encoder. We pretrain the encoder by training the VRNN on all source and target domain samples for a desired source-target adaptation pair. In order to choose how many samples would be used for training, we looked at which domain had more examples and chose the larger of the two. For example, if the source domain was group 2 with 508 patients and the target domain was group 5 with 437 patients, the VRNN would see 508 samples of each domain, with group 5 being sampled with replacement after seeing all its samples. As the encoder was used for learning latent representations, we thought it worth investigating whether if pretrained it better captured the latent representations that were being used by the domain classifier for adversarial training. We thought beginning domain classification at a better initialization point might help VRADA avoid local minima. For each method, we fed one source domain sample to Gy and either a source or target domain sample to Gd. (For this training and all training samples, order was randomized.) We only calculated the loss Lr once for the Gd samples so as to not bias the optimization of the VRNN. Table 3 shows the results of AHRF Mortality Prediction task for different types of VRADA training. From these experiments, we found that jointly training VRADA (i.e method I) usually performed better than the other pretrained training approaches.
6.2 MODEL VARIATIONS
6.2.1 ADVERSARIAL TRAINING AT EVERY TIME-STEP
A natural question is whether adversarial training at every time-step is more effective than adversarial training at the last time-step of a latent representation. If done at every time-step, the network learns
to create domain-invariant representations of subsets of your input x≤T . Do these domain-invariant representations help the network find more optimal domain-invariant representations of x? We empirically tested this scenario (Table 4) and found the results to be sub-optimal when compared to only performing adversarial training at the last time-step (Table 1). Below are results for the R-DANN and VRADA models for adversarial training at every time-step.
6.2.2 EFFECT OF RECONSTRUCTION LOSS
Table 5 shows the effect of reconstruction loss for our VRADA model. We observe that reconstructing the original data (i.e. using the decoder for reconstructing the data) helps in the overall performance improvement of our VRADA model.
6.2.3 IMPACT OF ADVERSARIAL TRAINING
In figures 5 and 6 we show the cell state activations for the VRADA and R-DANN without domain adaptation (i.e. no adversarial training). From these figures, we see that the dependencies between source and target domains are not transferred correctly since we do not perform adversarial training. On the otherhand, as discussed in section 4.4, figure 3 shows that adversarial training helps in transferring the dependencies between source and target domains efficiently.
6.3 R-DANN MODEL INFORMATION
Here we provide more details on the network architectures of the R-DANN and DANN. Please refer to Figure 7 for a diagram of the R-DANN model showing the dimensions of each layer and the connections between layers. The R-DANN and DANN were essentially identical except that, for the DANN, the first layer used a fully-connected layer instead of an RNN and took input flattened over the time-dimension. Thus the input dimensions corresponded to f and t× f for the R-DANN and DANN, respectively, where f is the number of features and t is the length of the time-dimension.

Review:
The work combines variational recurrent neural networks, and adversarial neural networks to handle domain adaptation for time series data. The proposed method, along with several competing algorithms are compared on two healthcare datasets constructed from MIMIC-III in domain adaptation settings.
The new contribution of the work is relatively small. It extends VRNN with adversarial training for learning domain agnostic representations. From the experimental results, the proposed method clearly out-performs competing algorithms. However, it is not clear where the advantage is coming from. The only difference between the proposed method and R-DANN is using variational RNN vs RNN. Little insights were provided on how this could bring such a big difference in terms of performance and the drastic difference in the temporal dependencies captured by these two methods in Figure 4.  
Detailed comments:
1. Please provide more details on what is plotted in Figure 1. Is 1 (b) is the t-sne projection of representations learned by DANN or R-DANN? The text in section 4.4 suggests it’s the later case. It is surprising to see such a regular plot for VRADA.  What do you think are the two dominant latent factors encoded in figure 1 (c)? 
2. In Table 2, the two baselines have quite significant difference in performance testing on the entire target (including validation set) vs on the test set only. VRADA, on the other hand, performs almost identical in these two settings. Could you please offer some explanation on this?
3. Please explain figure 3 and 4 in more details. how to interpret the x-axis of figure 3, and the x and y axes of figure 4. Again the right two plots in figure 4 are extremely regular comparing to the ones on the left.