Original: I DENTIFYING THROUGH F LOWS FOR R ECOVERING L ATENT R EPRESENTATIONS
Revised: I DENTIFYING THROUGH F LOWS FOR R ECOVERING L ATENT R EPRESENTATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Identiﬁability, or recovery of the true latent representations from which the observed data originates, is  a fundamental goal of representation learning.
Revised: Identiﬁability, or recovery of the true latent representations from which the observed data originates, is de facto a fundamental goal of representation learning.

Original: However, most deep generative models do not address the question of identiﬁability, and cannot recover the true latent sources that generate the observations.
Revised: Yet, most deep generative models do not address the question of identiﬁability, and thus fail to deliver on the promise of the recovery of the true latent sources that generate the observations.

Original: Recent work proposed identiﬁable generative modelling using variational autoencoders (iVAE) with a theory of identiﬁability.
Revised: Recent work proposed identiﬁable generative modelling using variational autoencoders (iVAE) with a theory of identiﬁability.

Original: However, due to the intractablity of KL divergence between variational approximate posterior and the true posterior,  iVAE has to maximize the evidence lower bound  of the marginal likelihood, leading to suboptimal solutions in both theory and practice.
Revised: Due to the intractablity of KL divergence between variational approximate posterior and the true posterior, however, iVAE has to maximize the evidence lower bound (ELBO) of the marginal likelihood, leading to suboptimal solutions in both theory and practice.

Original: In contrast, we propose an identiﬁable framework for estimating latent representations using a ﬂow-based model (iFlow).
Revised: In contrast, we propose an identiﬁable framework for estimating latent representations using a ﬂow-based model (iFlow).

Original: Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on identiﬁability, without the need for variational approximations.
Revised: Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on identiﬁability, thereby dispensing with variational approximations.

Original: We derive its learning objective in analytical form, making it possible to train iFlow in an end-to-end manner.
Revised: We derive its optimization objective in analytical form, making it possible to train iFlow in an end-to-end manner.

Original: Simulations on synthetic data validate the correctness and effectiveness of our proposed method and demonstrate its practical advantages over other existing methods.
Revised: Simulations on synthetic data validate the correctness and effectiveness of our proposed method and demonstrate its practical advantages over other existing methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental question in representation learning relates to identiﬁability: when is it possible to recover the true latent representations that generate the observed data?
Revised: A fundamental question in representation learning relates to identiﬁability: under which condition is it possible to recover the true latent representations that generate the observed data?

Original: Most existing  approaches for deep generative modelling, such as Variational Autoencoders (VAE) (Kingma & Welling, 2013) and ﬂow-based methods (Kobyzev et al., 2019), focus on learning latent-variable distributions and generating realistic data samples, but do not address the question of identiﬁability, i.e. recovering the true latent representations.
Revised: Most existing likelihood-based approaches for deep generative modelling, such as Variational Autoencoders (VAE) (Kingma & Welling, 2013) and ﬂow-based models (Kobyzev et al., 2019), focus on performing latent-variable inference and efﬁcient data synthesis, but do not address the question of identiﬁability, i.e. recovering the true latent representations.

Original: The question of identiﬁability is closely related to the goal of learning disentangled representations (Bengio et al., 2013).
Revised: The question of identiﬁability is closely related to the goal of learning disentangled representations (Bengio et al., 2013).

Original: A disentangled representation is deﬁned as one where individual latent units are sensitive to changes in single generative factors, while being relatively invariant to nuisance factors (Bengio et al., 2013).
Revised: While there is no canonical deﬁnition for this term, we adopt the one where individual latent units are sensitive to changes in single generative factors while being relatively invariant to nuisance factors (Bengio et al., 2013).

Original: A good representation for human faces, for example, should encompass different latent factors that separately encode different characteristics including gender, hair color, facial expression, etc.
Revised: A good representation for human faces, for example, should encompass different latent factors that separately encode different attributes including gender, hair color, facial expression, etc.

Original: By aiming to recover the true latent representation, identiﬁable models also allow for principled disentanglement; this suggests that rather than being entangled in disentanglement learning in a completely unsupervised manner, we go a step further towards identiﬁability, since existing literature on disentangled representation learning, such as β -VAE (Higgins et al., 2017), β -TCVAE
Revised: By aiming to recover the true latent representation, identiﬁable models also allow for principled disentanglement; this suggests that rather than being entangled in disentanglement learning in a completely unsupervised manner, we go a step further towards identiﬁability, since existing literature on disentangled representation learning, such as β -VAE (Higgins et al., 2017), β -TCVAE

Original: (Chen et al., 2018), DIP-VAE (Kumar et al., 2017) and FactorVAE (Kim &
Revised: (Chen et al., 2018), DIP-VAE (Kumar et al., 2017) and FactorVAE (Kim &

Original: Mnih, 2018) are neither general endeavors to achieve identiﬁability, nor do they provide theoretical guarantees on recovering the true latent sources.
Revised: Mnih, 2018), are neither general endeavors to achieve identiﬁability; nor do they provide theoretical guarantees on recovering the true latent sources.

Original: Recently, Khemakhem et al.
Revised: Recently, Khemakhem et al.

Original: (2019) introduced a theory of identiﬁability for deep generative models, based upon which they proposed an identiﬁable variant of VAEs called iVAE, to learn the distribution over latent variables in an identiﬁable manner.
Revised: (2019) introduced a theory of identiﬁability for deep generative models, based upon which the authors proposed an identiﬁable variant of VAEs called iVAE, to learn the distribution over latent variables in an identiﬁable manner.

Original: However, the downside of learning such an identiﬁable model within the VAE framework lies in the intractability of KL divergence between the approximate posterior and the true posterior.
Revised: However, the downside of learning such an identiﬁable model within the VAE framework lies in the intractability of KL divergence between the approximate posterior and the true posterior.

Original: Therefore, in both theory and practice, iVAE inevitably leads to a suboptimal solution, which, rigorously speaking, renders the learned model unidentiﬁable.
Revised: Consequently, in both theory and practice, iVAE inevitably leads to a suboptimal solution, which renders the learned model far less identiﬁable.

Original: In this paper,  we propose to learn an identiﬁable generative model through ﬂows (short for normalizing ﬂows (Tabak et al., 2010; Rezende & Mohamed, 2015)).
Revised: In this paper, aiming at avoiding such a pitfall, we propose to learn an identiﬁable generative model through ﬂows (short for normalizing ﬂows (Tabak et al., 2010; Rezende & Mohamed, 2015)).

Original: A normalizing ﬂow is a transformation of a simple probability distribution (e.g. a standard normal) into a more complex probability distribution by a composition of a series of invertible and differentiable mappings (Kobyzev et al., 2019).
Revised: A normalizing ﬂow is a transformation of a simple probability distribution (e.g. a standard normal) into a more complex probability distribution by a composition of a series of invertible and differentiable mappings (Kobyzev et al., 2019).

Original: Hence, they can be exploited to effectively model complex probability distributions.
Revised: Hence, they can be exploited to effectively model complex probability distributions.

Original: In contrast to VAEs relying on variational approximations, ﬂow-based models allow for latent-variable inference and likelihood evaluation in an exact and efﬁcient manner, making them a perfect choice for  identiﬁability.
Revised: In contrast to VAEs relying on variational approximations, ﬂow-based models allow for latent-variable inference and likelihood evaluation in an exact and efﬁcient manner, making themselves a perfect choice for achieving identiﬁability.

Original: To this end, unifying identiﬁablity with ﬂows, we propose iFlow, a framework for deep latentvariable models which allows for recovery of the true latent representations from which the observed data originates.
Revised: To this end, unifying identiﬁablity with ﬂows, we propose iFlow, a framework for deep latentvariable models which allows for recovery of the true latent representations from which the observed data originates.

Original: We demonstrate that our ﬂow-based model makes it possible to directly maximize the conditional marginal likelihood and thus achieves identiﬁability in a rigorous manner.
Revised: We demonstrate that our ﬂow-based model makes it possible to directly maximize the conditional marginal likelihood and thus achieves identiﬁability in a rigorous manner.

Original: We provide theoretical guarantees on the recovery of the true latent representations, and show experiments on synthetic data to validate the theoretical and practical advantages of our proposed formulation over previous approaches.We will release our source code shortly.
Revised: We provide theoretical guarantees on the recovery of the true latent representations, and show experiments on synthetic data to validate the theoretical and practical advantages of our proposed formulation over prior approaches.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The objective of generative models is to model the data distribution, which can be arbitrarily complex.
Revised: An enduring demand in statistical machine learning is to develop probabilistic models that explain the generative process that produce observations. Oftentimes, this entails estimation of density that can be arbitrarily complex.

Original:  Normalizing Flows are a family of generative models that learns an invertible mapping between the observed data and certain latent variables over which a tractable distribution is deﬁned.
Revised: As one of the promising tools, Normalizing Flows are a family of generative models that ﬁt such a density of exquisite complexity by pushing an initial density (base distribution) through a series of transformations.

Original: Formally, let x ∈ X ⊆ R n be an observed random variable, and z ∈ Z ⊆ R n a latent variable with a tractable distribution.
Revised: Formally, let x ∈ X ⊆ R n be an observed random variable, and z ∈ Z ⊆ R n a latent variable obeying a base distribution p Z ( z ) .

Original: Let f be an invertible function such that x = f ( z ) .
Revised: A normalizing ﬂow f is a diffeomorphism (i.e an invertible differentiable transformation with differentiable inverse) between two topologically equivalent spaces X and Z such that x = f ( z ) .

Original: By using thechange of variable formula, the probability densityfunction (pdf) of x is given by 
Revised: Under these conditions, the density of x is well-deﬁned and can be obtained by using the change of variable formula:

Original: [Equation]
Revised: [Equation]

Original: where h is the inverse of f .
Revised: where h is the inverse of f .

Original: To approximate an arbitrarily complex nonlinear invertible bijection, we can compose a series of such functions, since the composition of invertible functions is also invertible, and its Jacobian determinant is the product of the individual functions’ Jacobian determinants.
Revised: To approximate an arbitrarily complex nonlinear invertible bijection, one can compose a series of such functions, since the composition of invertible functions is also invertible, and its Jacobian determinant is the product of the individual functions’ Jacobian determinants.

Original: Speciﬁcally, let f 1 , f 2 , ..., f L be a set of L invertible functions with their corresponding inverses h 1 , h 2 , ..., h L .

Original: Then, the probability density function (pdf) of x can be obtained by successively transforming z through a sequence of L invertible functions f l ’s:
Revised: Denote φ as the diffeomorphism’s learnable parameters. Optimization can proceed as follows by maximizing log-likelihood for the density estimation model:

Original: [Equation]
Revised: [Equation]

Original: where z l def = f l ◦ · · · ◦ f 1 ( z ) and z L def = x .
Revised: [Equation]

Original:  R ELATED W ORK Nonlinear ICA
Revised:  R ELATED W ORK Nonlinear ICA

Original: Nonlinear ICA is a fundamental task in unsupervised learning that has attracted a great amount of attention in recent years.
Revised: Nonlinear independent component analysis (ICA) is one of the biggest problems remaining unresolved in unsupervised learning.

Original: Given the observations alone, it aims to recover the inverse mixing function as well as their corresponding independent sources.
Revised: Given the observations alone, it aims to recover the inverse mixing function as well as their corresponding independent sources.

Original: In contrast with the linear case, research on nonlinear ICA is hampered by the fact that without auxiliary variables, recovering the independent latents is impossible (Hyv¨arinen & Pajunen, 1999).
Revised: In contrast with the linear case, research on nonlinear ICA is hampered by the fact that without auxiliary variables, recovering the independent latents is impossible (Hyv¨arinen & Pajunen, 1999).

Original: Similar impossibility result can be found in (Locatello et al., 2018).
Revised: Similar impossibility result can be found in (Locatello et al., 2018).

Original: Fortunately, by exploiting additional temporal structure on the sources, recent work (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2018) established the ﬁrst identiﬁability results for deep latent-variable models.
Revised: Fortunately, by exploiting additional temporal structure on the sources, recent work (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2018) established the ﬁrst identiﬁability results for deep latent-variable models.

Original: These approaches, however, do not explicitly learn the data distribution, nor are they capable of generating “fake” data.
Revised: These approaches, however, do not explicitly learn the data distribution; nor are they capable of generating “fake” data.

Original: Khemakhem et al.
Revised: Khemakhem et al.

Original: (2019) bridged this gap by establishing a principled connection between VAEs and an identiﬁable model for nonlinear ICA.
Revised: (2019) bridged this gap by establishing a principled connection between VAEs and an identiﬁable model for nonlinear ICA.

Original: Their method with an identiﬁable VAE (known as iVAE) approximates the true joint distribution over observed and latent variables under mild conditions.
Revised: Their method with an identiﬁable VAE (known as iVAE) approximates the true joint distribution over observed and latent variables under mild conditions.

Original: However, due to the intractablity of KL divergence between variational approximate posterior and the true posterior, iVAE maximizes the evidence lower bound on the data log-likelihood, which in both theory and practice inevitably leads to suboptimal identifying performance.
Revised: Nevertheless, due to the intractablity of KL divergence between variational approximate posterior and the true posterior, iVAE maximizes the evidence lower bound on the data log-likelihood, which in both theory and practice acts as a detriment to the achievement of identiﬁability.

Original: We instead propose identifying through ﬂows (normalizing ﬂow), which maximizes the likelihood in a straightforward way, providing theoretical guarantees and practical advantages for identiﬁability.
Revised: We instead propose identifying through ﬂows (normalizing ﬂow), which maximizes the likelihood in a straightforward way, providing theoretical guarantees and practical advantages for identiﬁability.

Original: Normalizing Flows
Revised: Normalizing Flows

Original: Normalizing Flows are a family of generative approaches that models a data distribution by learning a bijection from observations to latent codes, and vice versa.
Revised: Normalizing Flows are a family of generative approaches that ﬁts a data distribution by learning a bijection from observations to latent codes, and vice versa.

Original: Compared with VAEs which learn a posterior approximation to the true posterior, normalizing ﬂows directly deal with marginal likelihood with exact inference while maintaining efﬁcient sampling.
Revised: Compared with VAEs which learn a posterior approximation to the true posterior, normalizing ﬂows directly deal with marginal likelihood with exact inference while maintaining efﬁcient sampling.

Original: Formally, a normalizing ﬂow is a transform of a tractable probability distribution into a complex distribution by compositing a sequence of invertible and differentiable mappings.
Revised: Formally, a normalizing ﬂow is a transform of a tractable probability distribution into a complex distribution by compositing a sequence of invertible and differentiable mappings.

Original: In practice, the challenge lies in designing a normalizing ﬂow that satisﬁes the following conditions: (1) it should be bijective and thus invertible; (2) it is efﬁcient to compute its inverse and its Jacobian determinant while maintaining sufﬁcient capabilities.
Revised: In practice, the challenge lies in designing a normalizing ﬂow that satisﬁes the following conditions: (1) it should be bijective and thus invertible; (2) it is efﬁcient to compute its inverse and its Jacobian determinant while ensuring sufﬁcient capabilities.

Original: The framework of normalizing ﬂows was ﬁrst deﬁned in (Tabak et al., 2010) and (Tabak & Turner, 2013) and then explored for density estimation in (Rippel & Adams, 2013).
Revised: The framework of normalizing ﬂows was ﬁrst deﬁned in (Tabak et al., 2010) and (Tabak & Turner, 2013) and then explored for density estimation in (Rippel & Adams, 2013).

Original: Rezende & Mohamed (2015) applied normalizing ﬂows to variational inference by introducing planar and radial ﬂows.
Revised: Rezende & Mohamed (2015) applied normalizing ﬂows to variational inference by introducing planar and radial ﬂows.

Original: Since then, various ﬂows have been proposed.
Revised: Since then, there had been abundant literature towards expanding this family.

Original: Kingma & Dhariwal (2018) parameterizes linear ﬂows with the LU factorization and “ 1 × 1 ” convolutions for the sake of efﬁcient determinant calculation and invertibility of convolution operations.
Revised: Kingma & Dhariwal (2018) parameterizes linear ﬂows with the LU factorization and “ 1 × 1 ” convolutions for the sake of efﬁcient determinant calculation and invertibility of convolution operations.

Original: Despite their limits in expressive capabilities, linear ﬂows serve as essential building blocks of afﬁne coupling ﬂows as in (Dinh et al., 2014; 2016).
Revised: Despite their limits in expressive capabilities, linear ﬂows act as essential building blocks of afﬁne coupling ﬂows as in (Dinh et al., 2014; 2016).

Original: Kingma et al.
Revised: Kingma et al.

Original: (2016) applied autoregressive models as a form of normalizing ﬂows, which exhibit strong expressiveness in modelling statistical dependencies among variables.
Revised: (2016) applied autoregressive models as a form of normalizing ﬂows, which exhibit strong expressiveness in modelling statistical dependencies among variables.

Original: However, the forwarding operation of autoregressive models is inherently sequential, which makes it inefﬁcient for training.
Revised: However, the forwarding operation of autoregressive models is inherently sequential, which makes it inefﬁcient for training.

Original: Splines have also been used as building blocks of normalizing ﬂows: M ¨ uller et al.
Revised: Splines have also been used as building blocks of normalizing ﬂows: M¨uller et al.

Original: (2018) suggested modelling a linear and quadratic spline as the integral of a univariate monotonic function for ﬂow construction.
Revised: (2018) suggested modelling a linear and quadratic spline as the integral of a univariate monotonic function for ﬂow construction.

Original: Durkan et al. (2019a) proposed a natural extension to the framework of neural importance sampling and also suggested modelling a coupling layer as a monotonic rational-quadratic spine (Durkan et al., 2019b), which can be implemented either with a coupling architecture RQ-NSF(C) or with autoregressive architecture RQ-NSF(AR).
Revised: Durkan et al. (2019a) proposed a natural extension to the framework of neural importance sampling and also suggested modelling a coupling layer as a monotonic rational-quadratic spine (Durkan et al., 2019b), which can be implemented either with a coupling architecture RQ-NSF(C) or with autoregressive architecture RQ-NSF(AR).

Original: The expressive capabilities of normalizing ﬂows and their theoretical guarantee of invertibility make them a natural choice for recovering the true mixing mapping from sources to observations, and thus identiﬁability can be rigorously achieved.
Revised: The expressive capabilities of normalizing ﬂows and their theoretical guarantee of invertibility make them a natural choice for recovering the true mixing mapping from sources to observations, and thus identiﬁability can be rigorously achieved.

Original: In our work, we show that by introducing normalizing ﬂows  it is possible to learn an identiﬁable latent-variable model with theoretical guarantees of identiﬁability.
Revised: In our work, we show that by aligning normalizing ﬂows with an existing identiﬁability theory, it is desirable to learn an identiﬁable latent-variable model with theoretical guarantees of identiﬁability.

Original:  I DENTIFIABLE F LOW
Revised:  I DENTIFIABLE F LOW

Original: In this section, we ﬁrst introduce the identiﬁable latent-variable family and the theory of identiﬁability (Khemakhem et al., 2019) that makes it possible to recover the joint distribution between observations and latent variables.
Revised: In this section, we ﬁrst introduce the identiﬁable latent-variable family and the theory of identiﬁability (Khemakhem et al., 2019) that makes it possible to recover the joint distribution between observations and latent variables.

Original: Then we derive our model, iFlow, and its optimization objective which leads to principled disentanglement with theoretical guarantees of identiﬁability.
Revised: Then we derive our model, iFlow, and its optimization objective which admits principled disentanglement with theoretical guarantees of identiﬁability.

Original: 4. I DENTIFIABLE L ATENT - VARIABLE F AMILY
Revised: 4. I DENTIFIABLE L ATENT - VARIABLE F AMILY

Original: The primary assumption leading to identiﬁability is a conditionally factorized prior distribution over the latent variables, p θ ( z | u ) , where u is an auxiliary variable, which can be the time index in a time series, categorical label, or an additionally observed variable (Khemakhem et al., 2019).
Revised: The primary assumption leading to identiﬁability is a conditionally factorized prior distribution over the latent variables, p θ ( z | u ) , where u is an auxiliary variable, which can be the time index in a time series, categorical label, or an additionally observed variable (Khemakhem et al., 2019).

Original: Formally, let x ∈ X ⊆ R n and u ∈ U ⊆
Revised: Formally, let x ∈ X ⊆ R n and u ∈ U ⊆

Original: R m be two observed random variables, and z ∈ Z ⊆ R n a latent variable that is the source of x .
Revised: R m be two observed random variables, and z ∈ Z ⊆ R n a latent variable that is the source of x .

Original: This implies that there can be an arbitrarily complex nonlinear mapping f : Z → X .
Revised: This implies that there can be an arbitrarily complex nonlinear mapping f : Z → X .

Original: Assuming that f is a bijection, it is desirable to recover its inverse by approximating using a family of invertible mappings h φ parameterized by φ .
Revised: Assuming that f is a bijection, it is desirable to recover its inverse by approximating using a family of invertible mappings h φ parameterized by φ .

Original: The statistical dependencies among these random variables are deﬁned by a Bayesian net: u → z → x , from which the following conditional generative model can be derived:
Revised: The statistical dependencies among these random variables are deﬁned by a Bayesian net: u → z → x , from which the following conditional generative model can be derived:

Original: [Equation]
Revised: [Equation]

Original: where p ( x | z ; φ ) def = p (cid:15) ( x − h − 1 ( z )) and p ( z | u ; T , λ ) is assumed to be a factorized exponential family distribution conditioned upon u .
Revised: where p ( x | z ; φ ) def = p (cid:15) ( x − h − 1 ( z )) and p ( z | u ; T , λ ) is assumed to be a factorized exponential family distribution conditioned upon u .

Original: Note that this density assumption is valid in most cases, since the exponential families have universal approximation capabilities (Sriperumbudur et al., 2017).
Revised: Note that this density assumption is valid in most cases, since the exponential families have universal approximation capabilities (Sriperumbudur et al., 2017).

Original: Speciﬁcally, the probability density function is given by
Revised: Speciﬁcally, the probability density function is given by

Original: [Equation]
Revised: [Equation]

Original: where Q i is the base measure, Z i ( u ) is the normalizing constant, T i,j ’s are the components of the sufﬁcient statistic and λ i,j ( u ) the natural parameters, critically depending on u .
Revised: where Q i is the base measure, Z i ( u ) is the normalizing constant, T i,j ’s are the components of the sufﬁcient statistic and λ i,j ( u ) the natural parameters, critically depending on u .

Original: Note that k indicates the maximum order of statistics under consideration.
Revised: Note that k indicates the maximum order of statistics under consideration.

Original: 4. I DENTIFIABILITY T HEORY
Revised: 4. I DENTIFIABILITY T HEORY

Original: The objective of identiﬁability is to learn a model that is subject to (Khemakhem et al., 2019): foreach quadruplet (Θ , Θ
Revised: The objective of identiﬁability is to learn a model that is subject to: for

Original: [Equation]
Revised: [Equation]

Original: z
Revised: quadruplet

Original: [Equation]
Revised: [Equation]

Original: where Θ and Θ (cid:48) are two different choices of model parameters that imply the same marginal density.
Revised: where Θ and Θ (cid:48) are two different choices of model parameters that imply the same marginal density (Khemakhem et al., 2019).

Original: One possible way to achieve this objective is to introduce the deﬁnition of identiﬁability up to equivalence class:
Revised: One possible avenue towards this objective is to introduce the deﬁnition of identiﬁability up to equivalence class:

Original: Deﬁnition 4.1.
Revised: Deﬁnition 4.1.

Original: (Identiﬁability up to equivalence class) Let ∼ be an equivalence relation on Θ .
Revised: (Identiﬁability up to equivalence class) Let ∼ be an equivalence relation on Θ .

Original: A model deﬁned by p ( x , z ; Θ) =
Revised: A model deﬁned by p ( x , z ; Θ) =

Original: p ( x | z ; Θ) p ( z ; Θ) is said to be identiﬁable up to ∼ if
Revised: p ( x | z ; Θ) p ( z ; Θ) is said to be identiﬁable up to ∼ if

Original: [Equation]
Revised: [Equation]

Original: where such an equivalence relation in the identiﬁable latent-variable family is deﬁned as follows:
Revised: where such an equivalence relation in the identiﬁable latent-variable family is deﬁned as follows:

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: ( φ , ˜ T , ˜ λ ) and ( φ (cid:48) , ˜ T (cid:48) , ˜ λ (cid:48) ) are of the same equivalence class if and only if there exist A and c such that ∀ x ∈ X ,
Revised: ( φ , ˜ T , ˜ λ ) and ( φ (cid:48) , ˜ T (cid:48) , ˜ λ (cid:48) ) are of the same equivalence class if and only if there exist A and c such that ∀ x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: One can easily verify that ∼ is an equivalence relation by showing its reﬂexivity, symmetry and transitivity.
Revised: One can easily verify that ∼ is an equivalence relation by showing its reﬂexivity, symmetry and transitivity.

Original: Then, the identiﬁability of the latent-variable family is given by Theorem 4.1 (Khemakhem et al., 2019).
Revised: Then, the identiﬁability of the latent-variable family is given by Theorem 4.1 (Khemakhem et al., 2019).

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let Z
Revised: Let Z

Original: = Z 1 ×· · ·×Z n and suppose the following holds: (i)
Revised: = Z 1 ×· · ·×Z n and suppose the following holds: (i)

Original: The set { x ∈ X| Ψ (cid:15) ( x ) = 0 } has measure zero, where Ψ (cid:15) is the characteristic function of the density p (cid:15) ; (ii)
Revised: The set { x ∈ X| Ψ (cid:15) ( x ) = 0 } has measure zero, where Ψ (cid:15) is the characteristic function of the density p (cid:15) ; (ii)

Original: The sufﬁcient statistics T i,j in (2) are differentiable almost everywhere and ∂T i,j /∂z (cid:54) = 

Revised: The sufﬁcient statistics T i,j in (2) are differentiable almost everywhere and ∂T i,j /∂z (cid:54) =

Original: almost surely for z ∈
Revised: almost surely for z ∈

Original: Z
Revised: Z

Original: i and for all i ∈ { 1 , ..., n } and j ∈ { 1 , ..., k } .
Revised: i and for all i ∈ { 1 , ..., n } and j ∈ { 1 , ..., k } .

Original: (iii)
Revised: (iii)

Original: There exist ( nk +1 ) distinct priors u 0 , ..., u nk such that the matrix 
Revised: There exist ( nk +1 ) distinct priors u 0 , ..., u nk such that the matrix [Equation] 

Original: [Equation]
Revised: [Equation]

Original: of size
Revised: of size

Original: nk × nk is invertible.
Revised: nk × nk is invertible.

Original: Then, the parameters ( φ , ˜ T , ˜ λ ) are ∼ -identiﬁable.
Revised: Then, the parameters ( φ , ˜ T , ˜ λ ) are ∼ -identiﬁable.

Original: 4. O PTIMIZATION O BJECTIVE OF I F LOW
Revised: 4. O PTIMIZATION O BJECTIVE OF I F LOW

Original: We propose identifying through ﬂows (iFlow) for recovering latent representations.
Revised: We propose identifying through ﬂows (iFlow) for recovering latent representations.

Original: Our proposed model falls into the identiﬁable latent-variable family with (cid:15) = 0 , that is, p (cid:15) ( · ) = δ ( · ) , where δ is a point mass, i.e. Dirac measure.
Revised: Our proposed model falls into the identiﬁable latent-variable family with (cid:15) = 0 , that is, p (cid:15) ( · ) = δ ( · ) , where δ is a point mass, i.e. Dirac measure.

Original: Note that assumption (i) in Theorem 4.1 holds true for iFlow.
Revised: Note that assumption (i) in Theorem 4.1 holds true for iFlow.

Original: In stark contrast to iVAE which resorts to variational approximations and maximizes the evidence lower bound, iFlow directly maximizes the marginal likelihood conditioned on u :
Revised: In stark contrast to iVAE which resorts to variational approximations and maximizes the evidence lower bound, iFlow directly maximizes the marginal likelihood conditioned on u :

Original: [Equation]
Revised: [Equation]

Original: where p Z ( ·| u ) is modeled by a factorized exponential family distribution.
Revised: where p Z ( ·| u ) is modeled by a factorized exponential family distribution.

Original: Therefore, the log marginal likelihood is given by
Revised: Therefore, the log marginal likelihood is obtained:

Original: [Equation]
Revised: [Equation]

Original: where z i is the i th component of the source z
Revised: where z i is the i th component of the source z

Original: = h φ ( x ) [Equation] and T and λ are both n -by- k matrices.
Revised: = h φ ( x ) , and T and λ are both n -by- k matrices.

Original: Here, h φ is a normalizing ﬂow of any kind.
Revised: Here, h φ is a normalizing ﬂow of any kind.

Original: For the sake of simplicity, we set Q i ( z i )
Revised: For the sake of simplicity, we set Q i ( z i )

Original: = 

Revised: =

Original: for all i ’s and consider maximum order of sufﬁcient statistics of z i ’s up to 2, that is, k = 2 .
Revised: for all i ’s and consider maximum order of sufﬁcient statistics of z i ’s up to 2, that is, k = 2 .

Original: Hence, T and λ are given by
Revised: Hence, T and λ are given by

Original: [Equation]
Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the optimization objective is to minimize
Revised: the optimization objective is to minimize

Original: [Equation]
Revised: [Equation]

Original: trace

Original: [Equation]
Revised: [Equation]

Original: ) where p D denotes the empirical distribution, and the ﬁrst term in (10) is given by
Revised: where p D denotes the empirical distribution

Original: [Equation]
Revised: [Equation]

Original: In practice
Revised: and the ﬁrst term in (13) is given by n

Original: [Equation]
Revised: [Equation]

Original:  λ ( u ) can be modelled by a muli-layer perceptron with learnable parameters θ [Equation] where λ θ : R m → R 2 n .
Revised: In practice, λ ( u ) can be parameterized by a multi-layer perceptron with learnable parameters θ , where λ θ : R m → R 2 n .

Original: Here, m is the dimension of the space in which u ’s lies.
Revised: Here, m is the dimension of the space in which u ’s lies.

Original: Note that ξ i should be strictly negative in order for the exponential family’s probability density function to be ﬁnite.
Revised: Note that ξ i should be strictly negative in order for the exponential family’s probability density function to be ﬁnite.

Original: Negative softplus activation can be exploited to force this constraint.
Revised: Negative softplus nonlinearity can be exploited to force this constraint.

Original: Therefore,the optimization objective has the following closed-form to be optimized:
Revised: Therefore, optimization proceeds by minimizing the following closed-form objective:

Original: [Equation]
Revised: [Equation]

Original: λ θ ( u ) T (cid:19) − log (cid:12) (cid:12)(cid:12)(cid:12) det (cid:18) ∂ h φ ∂ x (cid:19)(cid:12) (cid:12)(cid:12)(cid:12)(cid:35) .
Revised: [Equation]

Original: (13) where Θ = { θ , φ } .
Revised: where Θ = { θ , φ } .

Original: 4. I DENTIFIABILITY OF I F LOW
Revised: 4. I DENTIFIABILITY OF I F LOW

Original: The identiﬁability of our proposed model, iFlow, is characterized by Theorem 4.2.
Revised: The identiﬁability of our proposed model, iFlow, is characterized by Theorem 4.2.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Minimizing L Θ with respect to Θ , in the limit of inﬁnite data, learns a model that is ∼ -identiﬁable.
Revised: Minimizing L Θ with respect to Θ , in the limit of inﬁnite data, learns a model that is ∼ -identiﬁable.

Original: Proof.
Revised: Proof.

Original: Minimizing L Θ with respect to Θ is equivalent to maximizing the log conditional likelihood, log p X ( x
Revised: Minimizing L Θ with respect to Θ is equivalent to maximizing the log conditional likelihood, log p X ( x

Original: | u ; Θ) .
Revised: | u ; Θ) .

Original: Given inﬁnite amount of data, maximizing log p X ( x | u ; Θ) will give us the true marginal likelihood conditioned on u , that is, p X ( x | u ; ˆΘ) =
Revised: Given inﬁnite amount of data, maximizing log p X ( x | u ; Θ) will give us the true marginal likelihood conditioned on u , that is, p X ( x | u ; ˆΘ) =

Original: p X ( x | u ; Θ ∗ ) , where ˆΘ = arg max Θ log p X ( x | u ; Θ) and Θ ∗ is the true parameter.
Revised: p X ( x | u ; Θ ∗ ) , where ˆΘ = arg max Θ log p X ( x | u ; Θ) and Θ ∗ is the true parameter.

Original: According to Theorem 4.1, we obtain that ˆΘ and Θ ∗ are of the same equivalence class deﬁned by ∼ .
Revised: According to Theorem 4.1, we obtain that ˆΘ and Θ ∗ are of the same equivalence class deﬁned by ∼ .

Original: Thus, according to Deﬁnition 4.1, the joint distribution parameterized by Θ is identiﬁable up to ∼ .
Revised: Thus, according to Deﬁnition 4.1, the joint distribution parameterized by Θ is identiﬁable up to ∼ .

Original: Consequently, Theorem 4.2 guaranteesa strong identiﬁablity of our proposed generative model, iFlow.
Revised: Consequently, Theorem 4.2 guarantees strong identiﬁability of our proposed generative model, iFlow.

Original: Note that unlike Theorem 3 in (Khemakhem et al., 2019), Theorem 4.2 makes no assumption that the family of approximate posterior distributions contains the true posterior.
Revised: Note that unlike Theorem 3 in (Khemakhem et al., 2019), Theorem 4.2 makes no assumption that the family of approximate posterior distributions contains the true posterior.

Original: And we show in experiments that this assumption is unlikely to hold true empirically.
Revised: And we show in experiments that this assumption is unlikely to hold true empirically.

Original:  S IMULATIONS
Revised:  S IMULATIONS

Original: To evaluate our method, we run simulations on a synthetic dataset.
Revised: To evaluate our method, we run simulations on a synthetic dataset.

Original: This section will elaborate on the details of the generated data set, implementation, evaluation metric and fair comparison withthe existing methods.
Revised: This section will elaborate on the details of the generated dataset, implementation, evaluation metric and fair comparison with existing methods.

Original: 5. D ATASET
Revised: 5. D ATASET

Original: We generate a synthetic dataset where the sources are non-stationary Gaussian time-series, as described in (Khemakhem et al., 2019): the sources are divided into M segments of L samples each.
Revised: We generate a synthetic dataset where the sources are non-stationary Gaussian time-series, as described in (Khemakhem et al., 2019): the sources are divided into M segments of L samples each.

Original: The auxiliary variable u is set to be the segment index.
Revised: The auxiliary variable u is set to be the segment index.

Original: For each segment, the conditional prior distribution is chosen from the exponential family (2), where k = 2 , Q i ( z i ) = 1 , and T i, 1 ( z i ) =
Revised: For each segment, the conditional prior distribution is chosen from the exponential family (4), where k = 2 , Q i ( z i ) = 1 , and T i, 1 ( z i ) =

Original: z 2 i ,
Revised: z 2 i ,

Original: T i, 2 ( z i ) =
Revised: T i, 2 ( z i ) =

Original: z i , and the true λ i,j ’s are randomly and independently generated across the segments and the components such that their variances obey a uniform distribution on [0 . 5 , 3] .
Revised: z i , and the true λ i,j ’s are randomly and independently generated across the segments and the components such that their variances obey a uniform distribution on [0 . 5 , 3] .

Original: The sources to recover are mixed by an invertible multi-layer perceptron (MLP) whose weight matrices are ensured to be full rank.
Revised: The sources to recover are mixed by an invertible multi-layer perceptron (MLP) whose weight matrices are ensured to be full rank.

Original: 5. I MPLEMENTATION D ETAILS
Revised: 5. I MPLEMENTATION D ETAILS

Original: The mapping λ θ that outputs the natural parameters of the conditional factorized exponential family is modeled by a multi-layer perceptron with the activation of the last layer being the softplus function.
Revised: The mapping λ θ that outputs the natural parameters of the conditional factorized exponential family is parameterized by a multi-layer perceptron with the activation of the last layer being the softplus nonlinearity.

Original: Additionally, a negative activation is taken on the second-order natural parameters in order to ensure the density to be ﬁnite.
Revised: Additionally, a negative activation is taken on the second-order natural parameters in order to ensure its ﬁniteness.

Original: The bijection h φ is modeled by RQ-NSF(AR) (Durkan et al., 2019b) with the ﬂow length of 10 and the bin 8, which gives rise to sufﬁcient ﬂexibility and expressiveness.
Revised: The bijection h φ is modeled by RQ-NSF(AR) (Durkan et al., 2019b) with the ﬂow length of 10 and the bin 8, which gives rise to sufﬁcient ﬂexibility and expressiveness.

Original: For each training iteration, we use a mini-batch of size 64, and an Adam optimizer with learning rate chosen in { 0 . 01 , 0 . 001 } to optimize the learning objective (12).
Revised: For each training iteration, we use a mini-batch of size 64, and an Adam optimizer with learning rate chosen in { 0 . 01 , 0 . 001 } to optimize the learning objective (15).

Original: 5. E VALUATION M ETRIC
Revised: 5. E VALUATION M ETRIC

Original: As a standard measure used in ICA, the mean correlation coefﬁcient (MCC) between the original sources and the corresponding predicted latents is chosen to be the evaluation metric.
Revised: As a standard measure used in ICA, the mean correlation coefﬁcient (MCC) between the original sources and the corresponding predicted latents is chosen to be the evaluation metric.

Original: A high MCC indicates the strong correlation between the identiﬁed latents recovered and the true sources.
Revised: A high MCC indicates the strong correlation between the identiﬁed latents recovered and the true sources.

Original: In experiments, we found that such a metric can be sensitive to the synthetic data generated by different random seeds.
Revised: In experiments, we found that such a metric can be sensitive to the synthetic data generated by different random seeds.

Original: We argue that unless one speciﬁes the overall generating procedure including random seeds in particular any comparison remains debatable.
Revised: We argue that unless one speciﬁes the overall generating procedure including random seeds in particular any comparison remains debatable.

Original: This is crucially important since most of the existing works failed to do so.
Revised: This is crucially important since most of the existing works failed to do so.

Original: Therefore, we run each simulation of different methods through seed 1 to seed 100 and report averaged MCCs with standard deviations, which makes the comparison fair and meaningful. 
Revised: Therefore, we run each simulation of different methods through seed 1 to seed 100 and report averaged MCCs with standard deviations, which makes the comparison fair and meaningful. [Figure]

Original: 5. C OMPARISON AND R ESULTS
Revised: 5. C OMPARISON AND R ESULTS

Original: We compare our model, iFlow, with iVAE.
Revised: We compare our model, iFlow, with iVAE.

Original: These two models are trained on the sameaforementioned synthetic dataset, with M = 40 , L = 1000 , n = d = 5 .
Revised: These two models are trained on the same synthetic dataset aforementioned, with M = 40 , L = 1000 , n = d = 5 .

Original: For visualization, we also apply another setting with M = 40 , L = 1000 , n = d = 2 .
Revised: For visualization, we also apply another setting with M = 40 , L = 1000 , n = d = 2 .

Original: To evaluate iVAE’s identifying performance, we use the original implementation that is ofﬁcially released1 with exactly the same settings as described in (Khemakhem et al., 2019)
Revised: To evaluate iVAE’s identifying performance, we use the original implementation that is ofﬁcially released with exactly the same settings as described in (Khemakhem et al., 2019)

Original: (cf.
Revised: (cf.

Original: Appendix A.2).
Revised: Appendix A.2).

Original: First, we demonstrate a visualization of identiﬁablity of these two models in a 2-D case ( n = d = 2 ) as illustrated in Figure 1, in which we plot the original sources (latent), observations and the identiﬁed sources recovered by iFlow and iVAE, respectively.
Revised: First, we demonstrate a visualization of identiﬁablity of these two models in a 2-D case ( n = d = 2 ) as illustrated in Figure 1, in which we plot the original sources (latent), observations and the identiﬁed sources recovered by iFlow and iVAE, respectively.

Original: Segments are marked with different colors.
Revised: Segments are marked with different colors.

Original: Clearly, iFlow outperforms iVAE in identifying the original sources while maintaining the[Figure] [Figure] original geometry of source manifold.
Revised: Clearly, iFlow outperforms iVAE in identifying the original sources while preserving the original geometry of source manifold.

Original: It is evident that the learned prior of iFlow bears much higher resemblance to the generating prior than that of iVAE in the presence of some trivial indeterminacies of scaling, global sign and permutation of the original sources, which are inevitable even in some cases of linear ICA.
Revised: It is evident that the latents recovered by iFlow bears much higher resemblance to the true latent sources than those by iVAE in the presence of some trivial indeterminacies of scaling, global sign and permutation of the original sources, which are inevitable even in some cases of linear ICA.

Original: This exhibits consistency with the deﬁnition of identiﬁability up to equivalence class that allows for existence of an afﬁne transformation between sufﬁcient statistics, as described in Proposition 4.1.
Revised: This exhibits consistency with the deﬁnition of identiﬁability up to equivalence class that allows for existence of an afﬁne transformation between sufﬁcient statistics, as described in Proposition 4.1.

Original: As shown in Figure 1(a), 1(c), and 1(d), iVAE achieves inferior identifying performance in the sense that its estimated priors tend to retain the manifold of the observations.
Revised: As shown in Figure 1(a), 1(c), and 1(d), iVAE achieves inferior identifying performance in the sense that its estimated latents tend to retain the manifold of the observations.

Original: Notably, we also ﬁnd that despite the relatively high MCC performance of iVAE in Figure 1(d), iFlow is much more likely to recover the true geometric manifold in which the latent sources lie.
Revised: Notably, we also ﬁnd that despite the relatively high MCC performance of iVAE in Figure 1(d), iFlow is much more likely to recover the true geometric manifold in which the latent sources lie.

Original: In Figure 1(b), iVAE’s estimated prior collapses in face of a highly nonlinearly mixing case, while iFlow still works well in identifying the  sources.
Revised: In Figure 1(b), iVAE’s recovered latents collapses in face of a highly nonlinearly mixing case, while iFlow still works well in identifying the true sources.

Original: Note that these are not rare occurrences.
Revised: Note that these are not rare occurrences.

Original: More visualization examples can be found in Appendix A.3.
Revised: More visualization examples can be found in Appendix A.3.

Original:  Second, regarding quantitative results as shown in Figure 2(a), our model, iFlow, consistently outperforms iVAE in MCC by a considerable margin across different random seeds under consideration while experiencing less uncertainty (standard deviation as indicated in the brackets).
Revised: https://github.com/ilkhem/iVAE/ [Figure] Second, regarding quantitative results as shown in Figure 2(a), our model, iFlow, consistently outperforms iVAE in MCC by a considerable margin across different random seeds under consideration while experiencing less uncertainty (standard deviation as indicated in the brackets).

Original: Moreover, Figure 2(b) also demonstrates that the energy value of iFlow is much higher than that of iVAE, which serves as evidence that the optimization of the evidence lower bound, as in iVAE, would lead to suboptimal identiﬁability.
Revised: Moreover, Figure 2(b) also showcases that the energy value of iFlow is much higher than that of iVAE, which serves as evidence that the optimization of the evidence lower bound, as in iVAE, would lead to suboptimal identiﬁability.

Original: The gap between the evidence lower bound and the conditional marginal likelihood is inevitably far from being negligible in practice.
Revised: As is borne out empirically, the gap between the evidence lower bound and the conditional marginal likelihood is inevitably far from being negligible in practice.

Original: For clearer analysis, we also report the correlation coefﬁcients for each source-latent pair in each dimension.
Revised: For clearer analysis, we also report the correlation coefﬁcients for each source-latent pair in each dimension.

Original: As shown in Figure 3, iFlow exhibits much stronger correlation than does iVAE in each single dimension of the latent space.
Revised: As shown in Figure 3, iFlow exhibits much stronger correlation than does iVAE in each single dimension of the latent space.

Original: Finally, we investigate the impact of different choices of activation for generating natural parameters of the exponential family distribution (see Appendix A.1 for details).
Revised: Finally, we investigate the impact of different choices of activation for generating natural parameters of the exponential family distribution (see Appendix A.1 for details).

Original: All of these choices are valid since theoretically the natural parameters form a convex space.
Revised: Note that all of these choices are valid since theoretically the natural parameters form a convex space.

Original: However, iFlow(Softplus) achieves the highest identifying performance, suggesting that the range of softplus allows for greater ﬂexibility, which makes itself a perfect choice for our network design.
Revised: However, iFlow(Softplus) achieves the highest identifying performance, suggesting that the range of softplus allows for greater ﬂexibility, which makes itself a good choice for natural parameter nonlinearity.

Original:  C ONCLUSION
Revised:  C ONCLUSIONS

Original: Among the most signiﬁcant goals of unsupervised learning is to learn the disentangled representations of observed data, or to identify original latent codes that generate observations (i.e. identiﬁability).
Revised: Among the most signiﬁcant goals of unsupervised learning is to learn the disentangled representations of observed data, or to identify original latent codes that generate observations (i e. identiﬁability).

Original: Bridging the theoretical and practical gap of rigorous identiﬁability, we propose to identify through ﬂows, which directly maximizes the marginal likelihood conditioned on auxiliary variables, establishing a natural framework for recovering original independent sources.
Revised: Bridging the theoretical and practical gap of rigorous identiﬁability, we have proposed iFlow, which directly maximizes the marginal likelihood conditioned on auxiliary variables, establishing a natural framework for recovering original independent sources.

Original: In theory, our contribution provides a rigorous proof of identiﬁability and hence the recovery of the joint distribution between observed and latent variables that leads to principled disentanglement.
Revised: In theory, our contribution provides a rigorous way to achieve identiﬁability and hence the recovery of the joint distribution between observed and latent variables that leads to principled disentanglement.

Original: Empirically, our approach also shows practical advantages over previous methods.
Revised: Extensive empirical studies conﬁrm our theory and showcase its practical advantages over previous methods

Original: A.
Revised: A CKNOWLEDGMENTS

Revised: This work is in loving memory of Kobe Bryant (1979-2020) ... We thank Xiaoyi Yin for helpful initial discussions

Original: A BLATION S TUDY ON A CTIVATIONS FOR N ATURAL P ARAMETERS
Revised: A BLATION S TUDY ON A CTIVATIONS FOR N ATURAL P ARAMETERS

Original: Figure A.1 demonstrates the comparison of MCC of iFlows implemented with different nonlinear activations for natural parameters and that of iVAE, in which relu+eps denotes the ReLU activation added by a small value (e.g. 1e-5) and sigmoid × 5 denotes the Sigmoid activation multiplied by 5.
Revised: Figure 4 demonstrates the comparison of MCC of iFlows implemented with different nonlinear activations for natural parameters and that of iVAE, in which relu+eps denotes the ReLU activation added by a small value (e.g. 1e-5) and sigmoid × 5 denotes the Sigmoid activation multiplied by 5.

Original: A. I MPLEMENTATION D ETAILS OF I VAE
Revised: A. I MPLEMENTATION D ETAILS OF I VAE

Original: As stated in Section 5.4, to evaluate iVAEs identifying performance, we use the original implementation that is ofﬁcially released with the same settings as described in (Khemakhem et al., 2019).
Revised: As stated in Section 5.4, to evaluate iVAE’s identifying performance, we use the original implementation that is ofﬁcially released with the same settings as described in (Khemakhem et al., 2019).

Original: Speciﬁcally, in terms of hyperparameters of iVAE, the functional parameters of the decoder and the inference model, as well as the conditional prior are chosen to be MLPs, where the dimension of the hidden layers is chosen in 50, 100, 200, the activation function is a leaky RELU or a leaky hyperbolic tangent, and the number of layers is chosen in 3, 4, 5, 6.
Revised: Speciﬁcally, in terms of hyperparameters of iVAE, the functional parameters of the decoder and the inference model, as well as the conditional prior are parameterized by MLPs, where the dimension of the hidden layers is chosen from { 50, 100, 200 } , the activation function is a leaky RELU or a leaky hyperbolic tangent, and the number of layers is chosen from { 3, 4, 5, 6 } .

Original: Here we report all averaged MCC scores of different implementations for iVAE as shown in Table 1.
Revised: Here we report all averaged MCC scores of different implementations for iVAE as shown in Table 1.

Original: Table 1 indicates that adding more layers or more hidden neurons does not improve MCC score, precluding the possibility that expressive capability is not the culprit of iVAE inferior performance.
Revised: Table 1 indicates that adding more layers or more hidden neurons does not improve MCC score, precluding the possibility that expressive capability is not the culprit of iVAE inferior performance.

Original: Instead, we argue that the assumption (i) of Theorem 3 in (Khemakhem et al., 2019) (i.e the family of approximate posterior distributions contains the true posterior) often fails or is hard to satisfy in practice, which is one of the major reasons for the inferior performance of iVAE.
Revised: Instead, we argue that the assumption (i) of Theorem 3 in (Khemakhem et al., 2019) (i.e the family of approximate posterior distributions contains the true posterior) often fails or is hard to satisfy in practice, which is one of the major reasons for the inferior performance of iVAE.

Original: Additionally,
Revised: Additionally,

Original: Figure 2(b) demonstrates that the energy value of iFlow is much higher than that of iVAE, which provides evidence that optimizing the evidence lower bound, as in iVAE, leads to suboptimal identiﬁability.
Revised: Figure 2(b) demonstrates that the energy value of iFlow is much higher than that of iVAE, which provides evidence that optimizing the evidence lower bound, as in iVAE, leads to suboptimal identiﬁability.

Original: [Table]
Revised: [Table]

Original: A. V ISUALIZATION OF 2D C ASES
Revised: A. V ISUALIZATION OF 2D C ASES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]