Paper:
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
Intelligent behaviour in the real-world requires the ability to acquire new knowledge from an ongoing sequence of experiences while preserving and reusing past knowledge. We propose a novel algorithm for unsupervised representation learning from piece-wise stationary visual data: Variational Autoencoder with Shared Embeddings (VASE). Based on the Minimum Description Length principle, VASE automatically detects shifts in the data distribution and allocates spare representational capacity to new knowledge, while simultaneously protecting previously learnt representations from catastrophic forgetting. Our approach encourages the learnt representations to be disentangled, which imparts a number of desirable properties: VASE can deal sensibly with ambiguous inputs, it can enhance its own representations through imagination-based exploration, and most importantly, it exhibits semantically meaningful sharing of latents between different datasets. Compared to baselines with entangled representations, our approach is able to reason beyond surface-level statistics and perform semantically meaningful cross-domain inference.

Review:
By defining a generative process of images based on latent environments and shared latent generative factors which will be chosen depending on the environment, the authors propose an algorithm for life-long unsupervised representation learning. Reasonable inference procedures for the essential elements (the generative factors, the latent mask, and the environment) are proposed, and the training is done with a loss inspired in the Minimum Description Length principle and a "dreaming" feedback loop similar to the one in Shin et al. (2017).  The algorithm is a mix of several components for which the authors well state the motivations for them to be there. The work is mostly one of modeling, and no theoretical evidence supports the claims. All the evidence is experimental, which is very dense as a consequence of the several components and claims. Even though this work is of high quality, the quality comes from the engineering part. The authors proposed to tackle several scientific questions at once by mixing several components. In the end, the algorithm works and improves the state of the art, but there is not a clear principle that has been studied. However, this algorithm might be a good starting point to focus on one question such as to which extent the algorithm exhibits "semantically" meaningful sharing of latents between different datasets? Of course, one part of the experimental section is devoted to showing evidence for this question, but since there are several other claims, the evidence is superficial, there is no an attempt to falsify the claims to see if they are robust or at least to show the weaknesses of the algorithm. Doing this for each one of the claims would probably result in a paper with many more pages; this is why it would be desirable to focus on one specific question and answer it thoroughly. Studying better one aspect would increase the scientific quality of the paper, and improve the significance of the article.  The paper is written clearly, except for a few phrases which are not well explained I believe. For instance, in line 169, what motivates the use of the Wasserstein distance for the encoder and the KL divergence for the decoder? And the other examples are mostly about interpretation. To show clearly the evidence and the claims supported it might be a better idea to use words whose meaning in the given context is not up to ambiguity. For example, in line 268, "VASE can imagine the existence of moving MNIST before actually experiencing it" this phrase corresponds to an interpretation of the fact that when training a classifier for moving MNIST during the static MNIST training stage, the classifier can achieve an accuracy that is not random. The reasons that explain this fact might not have anything to do with "imagination" in the sense of artificially generated; maybe they could come from a translation invariant property of the classifier. The translation invariant explanation might not be correct, it is just an example to make a point of avoiding interpretations that implicitly make claims not supported by evidence, or at least for which the evidence has not been clearly explained.  To my knowledge, the idea of considering latent variables that might be or might not be shared across different environments and the idea of considering latent environments are new and original. I believe there might be exciting potential in exploring the latent environments idea profoundly. In my view, it would be more significant to focus on this part of the paper, giving clear empirical evidence for the proposed claims without filling the experimental part densely trying to do everything at once.  Update: I still think that it is better to tackle one specific question at a time; however, I find the number of experiments and the different ideas valuable, even if it is not an in-depth study (in my opinion) but an illustration of the potential of the algorithm. Also, as explained by the authors in the rebuttal, I understand that the several experiments are heavily linked (as a consequence of the nature of the algorithm, which is a mix of several components) which implies that it is not straightforward to isolate one specific question. I hope that the rearrangements in the experimental section that the authors mention in their rebuttal will help to illustrate the studied points better. For all these reasons, I have changed my score from 4 to 6.