Paper:
A significant body of recent work has examined variational autoencoders as a powerful approach for tasks which involve modeling the distribution of complex data such as images and text. In this work, we present a framework for modeling multiple data sets which come from differing distributions but which share some common latent structure. By incorporating architectural constraints and using a mutual information regularized form of the variational objective, our method successfully models differing data populations while explicitly encouraging the isolation of the shared and private latent factors. This enables our model to learn useful shared structure across similar tasks and to disentangle cross-population representations in a weakly supervised way. We demonstrate the utility of our method on several applications including image denoising, sub-group discovery, and continual learning.

Review:
This paper introduces a novel model called "Cross-Population Variational Autoencoder (CPVAE), which is designed to model data from different distributions sharing some common structure. The proposed generative model utilizes both shared and private per-population latent variables. In order to restrict shared latent variables from "leaking" into private representations, the authors introduce an information-theoretic regularizer. This regularizer forces private population representations to: (a) maximize mutual information with input samples from their population, and (b) minimize mutual information with input samples from other populations. In other words, private representations are forced to be "meaningful" on the corresponding population alone.
Quality:
The paper is well-written. I find the proposed method to be quite interesting. The derivations appear to be correct except for possibly the cancellation in Eq. 5, which I originally missed.
Significance:
In my opinion, if sound, the approach discussed in this paper may lead to interesting practical applications and may inspire other methods based on similar ideas.
Originality:
Even though, as authors point out, there is a substantial amount of work in this field, I believe that their approach is novel and has its own merits.
Clarity:
The paper is well written and the material is presented with clarity. In my opinion, the only exception is Section 4.4, which could definitely benefit from a few additional sentences describing the training procedure in more detail. Right now I find it a bit confusing. It would appear that the shared encoder / decoder continue to be trained as new populations arrive. Would this mean that catastrophic forgetting can actually impact this shared representation? And if it changes by a sufficient degree, can it reduce the quality of the generated samples for older populations? If so, I think these points should be mentioned in the text.
Questions and suggestions:
Experiments described in the paper are sufficiently convincing, but there are a few questions that could potentially be better clarified in the paper.
1. After reviewing the text again and seeing the comment of Reviewer #1, I am also confused about the cancellation in  I_q(x_k; t_k) - I_q(x_{-k}; \tilde{t}_k). Is it not true that marginal distributions q_\phi(t) and q_\phi(\tilde{t}) in the KL divergence term in Eq. 5 are different? Unfortunately, the final optimization objective relies on the cancellation of these terms and if they do not cancel, the approach may not be theoretically justified despite producing interesting and compelling results. (This affected the final rating. I will be able to change the rating once this point is clarified.)
2. Another issue is related to the special case when there are several very similar populations. Consider, for example, a case when there are two nearly-identical populations out of many. Using very similar latent variables for two similar populations would be penalized by the regularizer (not too significantly though). I assume that depending on the embedding sizes and the value of alpha (which authors introduce in Section 2.3), the model would either choose to use shared latent variables to encode these populations, or would allow for two nearly-identical private latent representations to exist. I think this is a conceptually important special case that could be mentioned and possibly explained in the paper.
3. I think the paper would also benefit from a clarification regarding the "mixing" function g. Choosing this function to be a simple sum of arguments seems restricting and may be insufficient for some datasets. It does not appear to be the case, but are there any restrictions on g? Can it come from a parametrized function family with parameters being optimized during training?
4. I think the paper would benefit from a more detailed discussion in Section 4.4 (see above).