Paper:
Learning from only partially-observed data for imputation has been an active research area. Despite promising progress on unimodal data imputation (e.g., image in-painting), models designed for multimodal data imputation are far from satisfactory. In this paper, we propose variational selective autoencoders (VSAE) for this task. Different from previous works, our proposed VSAE learns only from partially-observed data. VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation. Evaluation on both synthetic high-dimensional and challenging lowdimensional multi-modality datasets shows significant improvement over the stateof-the-art data imputation models.

Review:
The paper proposes a novel training method for variational autoencoders that allows using partially-observed data with multiple modalities. A modality can be a whole block of features (e.g., a MNIST image) or just a single scalar feature. The probabilistic model contains a latent vector per modality. The key idea is to use two types of encoder networks: a unimodal encoder for every modality which is used when the modality is observed, and a shared multimodal encoder that is provided all the observed modalities and produces the latent vectors for the unobserved modalities. The whole latent vector is passed through a decoder that predicts the mask of observed modalities, and another decoder that predicts the actual values of all modalities. The “ground truth” values for the unobserved modalities are provided by sampling from the corresponding latent variables from the prior distribution once at some point of training.
While I like the premise of the paper, I feel that it needs more work. My main concern is that sampling the target values for the unobserved modalities from the prior would almost necessarily lead to blurry synthetic “ground truth” for these modalities, which in turn means that the model would produce underconfident predictions for them. The samples from MNIST in Figure 3 are indeed very blurry, supporting this. Furthermore, the claims of the model working for non-MCAR missingness are not substantiated by the experiments. I believe that the paper should currently be rejected, but I encourage the authors to revise the paper.
Pros:
* Generative modelling of partially observed data is a very important topic that would benefit from fresh ideas and new approaches
* I really like the idea of explicitly modelling the mask/missingness vector. I agree with the authors that this should help a lot with non completely random missingness.
Cons:
* The text is quite hard to read. There are many typos (see below). The text is over the 8 page limit, but I don’t think this is justified. For example, the paragraph around Eqn. (11) just says that the decoder takes in a concatenated latent vector. The MNIST+SVHN dataset setup is described in detail, yet there is no summary of the experimental results, which are presented in the appendix.
* The approach taken to train on partially-observed data is described in three sentences after the Eqn. (10). The non-observed dimensions are imputed by reconstructions from the prior from a partially trained model. I think that this is the crux of the paper that should be significantly expanded and experimentally validated. It is possible that due to this design choice the method would not produce sharper reconstructions than the original samples from the prior. Figures 3, 5 and 6 indeed show very blurry samples from the model. Furthermore, it is not obvious to me why these prior samples would be sensible at all, given that all modalities have independent latents by construction.
* The paper states multiple times that VAEAC [Ivanov et al., 2019] cannot handle partially missing data, but I don’t think this is true, since their missing features imputation experiment uses the setup of 50% truly missing features. The trick they use is adding “synthetic” missing features in addition to the real ones and only train on those. See Section 4.3.3 of that paper for more details.
* The paper states that “it can model the joint distribution of the data and the mask together and avoid limiting assumptions such as MCAR”. However, all experiments only show results in the MCAR setting, so the claim is not experimentally validated.
* The baselines in the experiments could be improved. First of all, the setup for the AE and VAE is not specified. Secondly, it would be good to include a GAN-based baseline such as GAIN, as well as some more classic feature imputation method, e.g. MICE or MissForest.
* The experiments do not demonstrate that the model learns a meaningful *conditional* distribution for the missing modalities, since the provided figures show just one sample per conditioning image.
Questions to the authors:
1. Could you comment on the differences in your setup in Section 4.1 compared to the VAEAC paper? I’ve noticed that the results you report for this method significantly differ from the original paper, e.g. for VAEAC on Phishing dataset you report PFC of 0.24, whereas the original paper reports 0.394; for Mushroom it’s 0.403 vs. 0.244. I’ve compared the experimental details yet couldn’t find any differences, for example the missing rate is 0.5 in both papers.
2. How do you explain that all methods have NRMSE > 1 on the Glass dataset (Table 1), meaning that they all most likely perform worse than a constant baseline?
Typos and minor comments:
* Contributions (1) and (2) should be merged together.
* Page 2: to literature -> to the literature
* Page 2: “This algorithm needs complete data during training cannot learn from partially-observed data only.”
* Equations (1, 2): z and \phi are not consistently boldfaced
* Equations (4, 5): you can save some space by only specifying the factorization (left column) and merging the two equations on one row
* Page 4, bottom: use Bernoulli distribution -> use factorized/independent Bernoulli distribution
* Page 5, bottom: the word “simply” is used twice
* Page 9: learn to useful -> learn useful
* Page 9: term is included -> term included
* Page 9: variable follows Bernoulli -> variable following Bernoulli
* Page 9: conditions on -> conditioning on