Paper:
In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treatment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not controlled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose Rate-Distortion Optimization guided autoencoder, in which the Jacobi matrix from real space to latent space has orthonormality. It is proved theoretically and experimentally that (i) the probability distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant; (ii) our model behaves as non-linear PCA, where energy of acquired latent space is concentrated on several principal components and the influence of each component can be evaluated quantitatively. Furthermore, to verify the usefulness on the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.

Review:
Summary of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.
Review tl;dr: weak reject, for three main reasons:
(i) While the existing literature around VAEs, beta-VAEs,  and Rate-Distortion theory is mentioned in the related work, the connections are not nearly discussed sufficiently. 
(ii) On top of (i), the derivation of their loss function and architecture is not sufficiently motivated. This is in astonishing contrast to 1.5 pages of main text and 8 pages of (much appreciated!) analysis of properties.
(iii) Given the paper is clearly related to existing approaches in the literature, the experiments would require a much more careful comparison to existing models. It remains unclear why an interested user should favor your model over conceptually simpler generative models with fewer hyperparameters.
Detailed review:
Nota bene: This review is a late reassignment. While I reviewed the paper to the best of my ability, time constraints did not allow me to review parts of the paper in depth.  I am open to reassess my review during the second stage.
Connection to prior art: As a probabilistic, neural autoencoding model, the connections to the family of VAE models are obvious. The loss function (eq. (4)) still looks very much like the ELBO, where the typical conditional log-likelihood was split into two distortion terms. How is this different from e.g. a beta-VAE? Particularly, what is the connection between the rate-distortion analysis of beta-VAE by Alemi et al. and yours? These things need to be discussed explicitly, with more than a sentence or two in the related work section.
A lesser, but still important omission in your discussion of prior work: The Jacobian of the generator has also been studied, even for the VAE, cf. e.g. [1]. I believe this deserves more attention in your assessment of prior art.
Motivation: You use two distortion terms: actual sample vs. undistorted reconstruction. Why is that? What is the interpretation of the multipliers? How do I choose them? Why is a large part of your architecture (the pipeline from x to \hat(x)) actually deterministic? Why are you using the entropy of the prior over the latents, rather than the KL divergence between encoder and a prior? I think an interested reader could learn much more from your paper if you discussed your model embedded in th related work rather than in isolation.
Theory: Due to aforementioned time constraints, I was not able to review the extensive theoretical analysis in depth. Still, I would strongly recommend structuring the respective sections more clearly. Separate model and architecture description from the theoretical analysis; precisely formulate your claims. In particular, state your assumptions clearly. For instance, you assume "that each function's parameter is rich enough to fit ideally" (and similar e.g. in Appendix A). Does this only mean that the true distributions are part of the parametric family? What if this is not the case? Do your parameters need to be in the optimum for your analysis to hold true? 
Given that the full 20-page manuscript spends 10 pages on theory, I think this contribution is not given appropriate space in the main text.
Experiments: There are three experiments: a simple 3D proof of concept; anomaly detection; analysis of the latent state in CelebA. As mentioned in my review of the methods section, I believe the approach to be very similar to established models. None of the experiments provides convincing evidence why I should prefer the new, arguably more complex model.
For instance, I would have much preferred that you investigate properties of your model against alternatives over the anomaly detection experiments, which did not further my understanding of the proposed model. 
Summary: The paper tackles an important problem, namely the lack of control over the latent embedding in autoencoding generative models. I believe the author's contribution can be valuable, and I particularly appreciate the effort to investigate theoretical properties. As is, the case is not sufficiently convincing to be accepted, but I encourage the authors to improve the paper.
Minor comments:
1. While I appreciate a pun, I would recommend to rename the model along with the acronym to a more concise name.
2. Please revise your notation and typsetting. Examples: x1 instead of x_1, f of f(\cdot) instead of f(), \log instead of log.
3. Introduce acronyms before using them (e.g. VAE, MSE, SSIM), even when they seem obvious to you.
4. Please carefully check the manuscript for typos, missing articles, missing spaces etc.
5. Your citations are inconsistent, in that they sometimes use first names, sometimes first name initials, and sometimes no first names.
6. To my knowledge, the term scale function does not have an obvious definition. I think you are simply referring to monotonically increasing functions. Please clarify!
7. Your figures should be understandable without too much context, they need more detailed captions.
[1] http://proceedings.mlr.press/v84/chen18e.html