Paper:
It has been shown that using geometric spaces with non-zero curvature instead of plain Euclidean spaces with zero curvature improves performance on a range of Machine Learning tasks for learning representations. Recent work has leveraged these geometries to learn latent variable models like Variational Autoencoders (VAEs) in spherical and hyperbolic spaces with constant curvature. While these approaches work well on particular kinds of data that they were designed for e.g. tree-like data for a hyperbolic VAE, there exists no generic approach unifying all three models. We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature can be learned. This generalizes the Euclidean VAE to curved latent spaces, as the model essentially reduces to the Euclidean VAE if curvatures of all latent space components go to 0.

Review:
Summary: 
This paper is about developing VAEs in non-Euclidean spaces. Fairly recently, ML researchers have developed non-Euclidean embeddings, initially in hyperbolic space (constant negative curvature), and then in product spaces that have varying curvatures. These ideas were developed for embeddings, and recent attempts have been made to build entire models that operate in non-Euclidean spaces. The authors develop VAEs for the product spaces case.
There's largely two aspects here: one is to be able to write down the equivalents for the operations in models (e.g., the equivalent of adding or multiplying matrices and vectors in Euclidean space have to be lifted to other spaces which no longer have a linear structure). The other are VAE-specific choices, particularly choosing a normal distribution on the manifolds. The authors consider several of these choices and then run a variety of experiments on small latent-dimension cases for VAEs. These reveal that sometimes non-Euclidean and in particular product spaces improve performance.
Strengths, Weakness, Recommendation
I like what the authors are trying to do here; embeddings and discriminative models on non-Euclidean spaces have been developed, offer credible benefits, and generative models are the next step. The authors push forward the machinery needed to do this, and the results seem like there's something there.
On the other hand, the entire work seems quite preliminary. It's hard to say what the takeaway is, or any suggestions for users. The paper is written in a pretty frustrating way. There's an enormous amount of stuff in a sprawling appendix (there are 43 results in the first appendix?!), and checking all of these details will take a great deal of time. 
Overall, I recommended weak accept, since a lot of these issues seem like they can be cleaned up.
EDIT: I increased my score based on the authors' response.
Comments:
- The approach taken here is quite similar to another ICLR submission this year, which basically does the same thing but applies these operations to GCNs instead of VAEs.
- A better way to define curvature is just to talk about the sectional curvature, instead of the Gaussian curvature the authors mention at the beginning of section 2. Fortunately for the constant case all of these definitions will be the same.
- It's not quite clear in Section 2.1 why we should care about the fact that you can't fully take K->0 there---why does this hurt anything? You can approximate flat curvature arbitrarily well even without K exactly 0.
- On a similar theme, what's the point of doing the product of {E,S,D,H,P}, instead of just {E,S,H} or {E,D,P}? Seems a bit weird to consider all 5, given the equivalence between S-D and H-P.
- In 2.3, the products of spaces section, the distance decomposition in the 2nd paragraph should have squares (it's an l2): d_M(x,y)^2 = \sum_{i=1}^k d_{M_k_i^n_i)^2(x^i,y^i).
- The discussion in 2.3 should be expanded and made more concrete (some of these you can write out the expressions for), and more pros and cons explained, e.g., which theoretical properties are lost for the wrapped distributions?
- On page 6, I don't understand the first problem with the learnable curvature approach. Why is there no gradient w.r.t to K? Isn't the idea that you'll write this thing as a piecewise function (presumably it's continuous, since that's why the authors built those models that deform to flat), and differentiate the whole thing? Why wouldn't there be a gradient at ELBO(K)? Is it not differentiable at K=0? That doesn't follow directly from just saying the curvature is 0.
-  What's the intuition for the component learning algorithm using 2 dimensions for each of the spaces?
- The experiment section was written in a way where I couldn't understand why the choices being made were there. Why 6 and 12 dimensions here? More clarity here would be great. Also, are there any other models to compare against for these datasets? I'm not a VAE expert; what do other models typically obtain in the authors' regime?