Paper:
BRITS: Bidirectional Recurrent Imputation for Time Series
Time series are ubiquitous in many classification/regression applications. However, the time series data in real applications may contain many missing values. Hence, given multiple (possibly correlated) time series data, it is important to fill in missing values and at the same time to predict their class labels. Existing imputation methods often impose strong assumptions of the underlying data generating process, such as linear dynamics in the state space. In this paper, we propose a novel method, called BRITS, based on recurrent neural networks for missing value imputation in time series data. Our proposed method directly learns the missing values in a bidirectional recurrent dynamical system, without any specific assumption. The imputed values are treated as variables of RNN graph and can be effectively updated during backpropagation. BRITS has three advantages: (a) it can handle multiple correlated missing values in time series; (b) it generalizes to time series with nonlinear dynamics underlying; (c) it provides a data-driven imputation procedure and applies to general settings with missing data. We evaluate our model on three real-world datasets, including an air quality dataset, a healthcare dataset, and a localization dataset for human activity. Experiments show that our model outperforms the state-of-the-art methods in both imputation and classification/regression.

Review:
Post-rebuttal comment: The authors responded to some of my concerns. In particular, they addressed my question about the utility of this model to do imputation independent of the supervision task. According to their response, the imputed data produced by BRITS is superior in terms of downstream classification performance than forward-filled data, demonstrating utility. If this experiment can be extended to include comparisons from the other imputation methods then the paper will be strengthened. On this assumption, I change my score from a 5 to 6.  Summary of work: They describe a RNN-based framework for performing imputation and simultaneous classification of time series. The RNNs are trained to predict the next (or previous) value of the time-series , with a consistency requirement between forward and backward RNNs. The architecture makes use of masking variables to denote missing values, and time deltas since the last observed value. The latter are used to decay parts of the hidden state, and to weight the relative contributions to the imputed value from other variables, or from the historical prediction of that variable. Since imputation is performed at the same time as classification, they report both imputation performance and AUROC on three real-world datasets.  Explanation of overall score: I think this approach is solid, but the analyses are slightly lacking, limiting the general interest and applicability of the approach.  Explanation of confidence score: I am reasonably familiar with the related work, and there isn’t much mathematics to check.  Quality:  * The proposed model and variants are evaluated on three datasets for imputation, and two for imputation+binary classification. * I like the analysis in figure 4/appendix C, this is a useful experiment. * That the “correlated recurrent imputation” helps much over the independent version ((B)RITS v. (B)RITS-I) for imputation is only evident on the healthcare dataset - in the other cases, the difference is of the order of 0.1 percentage points or so (and RITS-I confusingly even outperforms RITS on the Human Activity dataset). Depending on the nature of the missingness this doesn’t seem entirely surprising - the difference between the independent and correlated prediction will depend on: 1) how correlated the variables actually are, and 2) how many non-missing values are observed alongside the data-point to be imputed. It would be interesting to see if these factors could explain the relative performance of BRITS v. BRITS-I on the different datasets. * How is the performance of GRU-D for imputation measured if it doesn’t impute missing data explicitly? * In applied machine learning, imputation is usually done as a precursor step to e.g. a supervised learning problem. I’m glad to see that performance on subsequent tasks is included in this work, but I would like to see more detail here. As indicated by the relatively poor imputation performance of GRU-D (although this may be related to the fact that it doesn’t perform imputation, as per my previous question), but the high classification performance, it would be interesting to see more about how imputation accuracy actually impacts downstream classification. I think to do this in a fair way you would need to separate the imputation and classification tasks - get each model to first perform imputation on the dataset, and then train the same classifier architecture (e.g. a random forest) on each imputed version of the dataset, and compare performance. This would enable the imputation performed by the non-RNN baselines to be compared. You could then also compare how training the RNN to classify at the same time as impute (as is done already) presumably improves classification performance, and potentially changes the imputed data. My intuition here is that certain imputation strategies may appear poor by absolute error, but may actually be adequate for the types of downstream tasks considered. * Highlighting a sub-part of the previous point - I think it would be interesting independent of the above investigation to see how much of an effect the classification task has on the imputation performance - that is, to turn off L_out in the loss. It would be an interesting result (and perhaps a useful contribution to the ongoing discussion around unsupervised learning) to demonstrate that having a task can improve imputation performance - and what would be doubly interesting would be to show that the resulting imputed data is still useful for *other* tasks! * We see empirically that BRITS outperforms GRU-D and MRNN, and the use of feature correlations is given as an explanation, but (B)RITS-I also outperforms both models, so something else must be going on. Is there something else about the (B)RITS architecture that’s providing an advantage? * One question: is it feasible to run this model to perform imputation in real-time? * In summary, I think the experiments here are interesting, but there are many more potentially interesting analyses which could be run.  Clarity:  * I like how this paper was laid out - the RITS-I -> BRITS-I -> BRITS progression made it quite easy to see how the model is constructed. * I think it would be useful to highlight where data is missing at random, or not-at-random in these experiments. I think it’s not at random in the Air Quality dataset, but at random in the others? The descriptions (e.g. figure 1, example 1) make it sound like the primary form of missingness is missing in blocks, which is slightly misleading. * Slight inconsistency: in the GRU-D paper, the AUROC on mortality in the PhysioNet data is reported at 0.8424 ± 0.012, which would make it comparable to the performance of BRITS (0.850 ± 0.002), but in this submission it is reported at 0.828 ± 0.004. * Minor: I think that errors should not be neglected when choosing which result to put in boldface (see last two rows, second column in Table 2 - these results are within error of each other). * Regarding reproducibility, I greatly appreciate that the authors provided an anonymous code repository, however the details of how the other RNN models were trained are missing.  Originality:  * It would be helpful if the other (GRU-D, MRNN) methods were described in a little more detail to make clear which elements of this model are original.  Significance:  * Solving problems on time-series data with missing data is quite important, so this work is significant in that sense. * However, I feel the analyses are slightly limited, making it hard to conclude what exactly about this architecture is providing benefit, or how generally useful it is for imputation (for example, in cases where the classification problem is not known ahead of time, or the classification architecture will not also be an RNN).