Paper:
We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept "disentangled representations" used in supervised and unsupervised methods along three dimensionsinformativeness, separability and interpretability-which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability, and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.

Review:
Summary:
The paper presents a new set of metrics for evaluating disentangled representations in both supervised and unsupervised settings. Disentangled representations are evaluated along three dimensions: informativeness, separability, and interpretability. While previous work offers metrics for similar dimensions (e.g., (Eastwood & Williams, 2018)), the paper suggests that the metrics of the submission are superior to (Eastwood & Williams, 2018), based on a comparison between FactorVAE and Beta-VAE.
Strengths:
(1) The metrics of the paper are well motivated from an information-theoretic standpoint. While, in some cases, the metrics themselves are straight applications of information theory (e.g., informativeness metric == mutual information), the authors came up with new metrics when existing information-theoretic definitions could be fooled by, e.g., introducing noisy and independent latent representations z.
(2) Experimental results show that these metrics are better than previous metrics at discriminating between FactorVAE and Beta-VAE (in an experimental setting where the former is clearly superior to the latter).
(3) As opposed to much of prior work, these metrics do not require any training, which can be considered a plus as they do not require adaptation for each (sub)domain.
(4) The paper is overall well written and clear. I very much enjoyed reading it.
Weaknesses:
(1) My main concern with the paper, which makes me vote for “weak accept” instead of “accept”: The metrics of the paper are compared against previous metrics (Eastwood & Williams, 2018) on *only two* types of disentangled representations, namely FactorVAE and Beta-VAE, and in one experimental condition. (There is plenty of other experimental material in the appendix which also introduces AAE, but none of it seems to compare against previous *metrics*). I think comparing metrics on only two systems is somewhat problematic, and we don’t know how general the results are. I would have preferred to see FactorVAE and Beta-VAE evaluated with fewer hyperparameter choices to allow introducing more variational approaches. Could you (the authors) at least evaluate AAE against Eastwood & Williams too? This concern is not just mere quibbling, as the authors have shown (e.g., in Section 3.2) that specific edge cases can fool naïve metrics (e.g., by introducing noisy and independent latent variables), and there may be other edge cases that the authors have not considered. I think evaluating new metrics against previous work (i.e., metrics) on only two underlying systems is not very convincing. I acknowledge that the paper offers *lots* of experiments – the full pdf is 30 pages (!) – but I think some of the existing ones could go to leave space for evaluations using more underlying VAE/baselines/edge-case systems.
(2) The paper presents six metrics (MI, MISJED, WSEPIN, WINDIN, RMIG, JEMMIG) along three dimensions. While each individual metric makes sense, I feel the paper lacks a discussion section that ties these pieces together and suggests a way of using these metrics conjointly across the three dimensions towards building better-disentangled representations (the paper has a “discussion” section, but it is very short and actually more of a conclusion). The more metrics we have, the more chances each underlying VAE model can “win” on one of the metrics, which is not particularly enlightening. 
(3) The paper is not self-contained, and some parts are almost impossible to understand without familiarity with (Kim & Mnih, 2018) and (Higgins et al., 2017a). It uses technical terms of these papers without explanations (e.g., TC is not even spelled out it seems). Hyperparameters of these papers (e.g., beta, gamma) are used without explanation.
(4) There is no related work section. Such a section could, e.g., make what is borrowed from (Kim & Mnih, 2018; Higgins et al., 2017a) more understandable.
Overall, I think it is a nice paper that makes significant contributions to the problem of building better disentangles representations thanks to better evaluation metrics. The empirical support for some of the claims (e.g., superiority to (Eastwood & Williams, 2018)) is a bit weak, but other strengths mentioned above largely make up for that.
Minor comments:
The definitions of SEPIN@k and INDIN@k don’t seem quite right. The summation iterates over z_0, …, z_k-1, leaving off z_k, …, z_L-1, but the latter variables might contain some z’s with lowest mutual information with x. The way the ‘sorted’ function is written only has the effect of reordering z_0, …, z_{k-1} among themselves, never considering any of the z_k and above whatever their MI’s are, which is probably not what the authors meant. Perhaps ‘sorted’ is intended to both sort and rename z’s, but if z’s are indeed renamed I think this should be indicated mathematically otherwise the equation is wrong (e.g., (z’_1, …, z’_{L-1}) = sorted_by_MI(z_1, …, z_{L_1})).
Typos in WSEPIN and WINDIN definitions: “0 = 1” -> “i = 1”?
Figure 1: “Image”? The paper is written in general terms and doesn’t seem to assume otherwise x is an image representation (or does it?).
Section 3.4: “Table. 1” -> “Table 1”
Table 1 vs. title of Section 3.1: The author’s informativeness metric doesn’t have a name, but the section title contains “informativeness,” which could be confusing vs. “informativeness” in Table 1 which is a completely different metric.
Page 8, interpretability: “TC=10”. Do you mean “TC loss” here? I presume 10 is the value of a hyperparameter of the FactorVAE paper (gamma?), and not the actual value of the TC term. Same question for Figure 9 and later figures of the appendix.