Paper:
Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised disentangled model selection. Our approach, Unsupervised Disentanglement Ranking (UDR), leverages the recent theoretical results that explain why variational autoencoders disentangle (Rolinek et al., 2019), to quantify the quality of disentanglement by performing pairwise comparisons between trained model representations. We show that our approach performs comparably to the existing supervised alternatives across 5400 models from six state of the art unsupervised disentangled representation learning model classes. Furthermore, we show that the ranking produced by our approach correlates well with the final task performance on two different domains.

Review:
This paper addresses the problem of unsupervised model selection for disentangled representation learning. Based on the understanding of “why VAEs disentangle” [Burgess et al. 2017, Locatello et al. 2018, Mathieu et al. 2019, Rolinek et al. 2019], the authors adopt the assumption that disentangled representations are all alike (up to permutation and sign inverse) while entangled representations are different, and propose UDR method and its variants. Experimental results clearly show that UDR is a good approach for hyperparameter/model selection.
Overall, I think a reliable metric for model selection/evaluation is needed for the VAE-based disentangled representation learning. According to comprehensive experimental studies performed in this paper, UDR seems to be a potentially good choice.
However, I am not sure if very good disentangled representations must benefit (general) subsequent tasks, though the authors provide experimental evidence on fairness classification and data efficiency tasks. Actually, the data generation process in the real-world may consist of different generative factors that are not independent of each other. Though good disentangled representation provides good interpretability, it needs not to be better than entangled representation for concrete tasks. Specifically, for concrete supervised classification tasks, VAE with beta smaller than 1 (not towards disentanglement) might be the best (Alexander A. Alemi et al. 2017, Deep VIB).
Another concern is about the choice of some key “hyperparameters”.
For the KL divergence threshold in equation 3, you set it to be 0.01. It looks like the choice would control how much the UDR favors a “sparse representation map”. The larger the value, the few “informative dimensions” would be considered.
In supplementary material, you say that “uninformative latents typically have KL<<0.01 while informative latents have KL >>0.01”. Is this judgment based on “qualitative feeling”? For me, as you are contributing a ``quantitative measurement”, it is interesting and important to see how this threshold would generally affect UDR’s behavior in one (or more) datasets you have tried. 
Another hyperparameter I cared is P (number of models for pairwise comparison). In the paper, you validate the effect of P in the range [5,45]. How would P smaller than 5 affect UDR? According to Table 1, if I was using UDR, I’d rather using P>=20 (or at least 10) rather than 5.
Also, it seems to me P would grow up due to the size of factors that generate the data. Thus, I also have a little concern about the computation cost of the proposed metric (as also mentioned by the authors).
Others concerns:
-- As a heavy experimental paper, most experimental results are in supplementary material, while the authors spent a lot of time in the main text explaining the conclusions found in other papers.
-- To validate the fundamental assumption of UDR, the authors might consider to quantitatively validate that, disentangled representations learned by those approaches you used in the paper are almost the same (up to permutation and sign inverse).