Original: N ONLINEAR
Revised: N ONLINEAR

Original: ICA
Revised: ICA

Original: USING V OLUME -P RESERVING
Revised: USING V OLUME -P RESERVING

Original: T RANSFORMATIONS
Revised: T RANSFORMATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Nonlinear ICA is a fundamental problem in machine learning, aiming to identify the underlying independent components (sources) from data which is assumed to be a nonlinear function (mixing function) of these sources.
Revised: Nonlinear ICA is a fundamental problem in machine learning, aiming to identify the underlying independent components (sources) from data which is assumed to be a nonlinear function (mixing function) of these sources.

Original: Recent works prove that if the sources have some particular structures (e.g. temporal structure), they are theoretically identiﬁable even if the mixing function is arbitrary.
Revised: Recent works prove that if the sources have some particular structures (e.g. temporal structure), they are theoretically identiﬁable even if the mixing function is arbitrary.

Original: However, in many cases such restrictions on the sources are difﬁcult to satisfy or even verify, hence it inhibits the applicability of the proposed methods.
Revised: However, in many cases such restrictions on the sources are difﬁcult to satisfy or even verify, hence it inhibits the applicability of the proposed methods.

Original: Different from these works, we propose a general framework for nonlinear ICA, in which the mixing function is assumed to be a volume-preserving transformation, and meanwhile the conditions on the sources can be much looser.
Revised: Different from these works, we propose a general framework for nonlinear ICA, in which the mixing function is assumed to be a volume-preserving transformation, and meanwhile the conditions on the sources can be much looser.

Original: We provide an insightful proof of the identiﬁability of the proposed framework.
Revised: We provide an insightful proof of the identiﬁability of the proposed framework.

Original: We implement the framework by volume-preserving Flow-based models, and verify our theory by experiments on artiﬁcial data and synthesized images.
Revised: We implement the framework by volume-preserving ﬂow-based models, and verify our theory by experiments on artiﬁcial data and synthesized images.

Original: Moreover, results on real-world images indicate that our framework can disentangle interpretable features.
Revised: Moreover, results on real-world images indicate that our framework can disentangle interpretable features.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Independent component analysis (ICA) is one of the most fundamental problems in machine learning.
Revised: Independent component analysis (ICA) is one of the most fundamental problems in machine learning.

Original: The earlier works concentrate on linear ICA (Comon, 1994), in which the observed data is assumed to be a linear and invertible transformation (called “mixing function”) of several independent components (called “sources”), and the goal is to identify the independent components from data points.
Revised: The earlier works concentrate on linear ICA (Comon, 1994), in which the observed data is assumed to be a linear and invertible transformation (called “mixing function”) of several independent components (called “sources”), and the goal is to identify the independent components from data points.

Original: Recently, there are increasing interests in nonlinear ICA (Hyv ¨ arinen & Pajunen, 1999), in which the mixing function is generalized to be nonlinear.
Revised: Recently, there are increasing interests in nonlinear ICA (Hyv ¨ arinen & Pajunen, 1999), in which the mixing function is generalized to be nonlinear.

Original: This nonlinear problem is crucial, as it has been the theoretical foundation of many important tasks.
Revised: This nonlinear problem is crucial, as it is the theoretical foundation of many important tasks.

Original: For example, one task is disentanglement (Locatello et al., 2019; Sorrenson et al., 2020; Locatello et al., 2020), which aims at learning some explanatory factors of variation from data (Bengio et al., 2013) and hence facilitates the interpretability of representation learning as well as its downstream tasks (Peters et al., 2017; Lake et al., 2017).
Revised: For example, one task is disentanglement (Locatello et al., 2019; Sorrenson et al., 2020; Locatello et al., 2020), which aims at learning some explanatory factors of variation from data (Bengio et al., 2013) and hence facilitates the interpretability of representation learning as well as its downstream tasks (Peters et al., 2017; Lake et al., 2017).

Original: Another example is controlling the generation process in generative models using semantic factors (Karras et al., 2019; Abdal et al., 2021).
Revised: Another example is controlling the generation process in generative models using semantic factors (Karras et al., 2019; Abdal et al., 2021).

Original: These tasks essentially require their latent variables to be identiﬁable in highly nonlinear latent models.
Revised: These tasks essentially require their latent variables to be identiﬁable in highly nonlinear latent models.

Original: Otherwise the latent variables can be mixing functions of explanatory / semantic factors, hence are probably not explanatory or semantic.
Revised: Otherwise the latent variables can be mixing functions of explanatory / semantic factors, hence are probably not explanatory or semantic.

Original: The central problem in nonlinear ICA is  unidentiﬁability.
Revised: The central problem in nonlinear ICA is the unidentiﬁability.

Original: Speciﬁcally, if the observed data points are independent and identically distributed (i.i.d.
Revised: Speciﬁcally, if the observed data points are independent and identically distributed (i.i.d.

Original: in short), i.e. there is no temporal or similar structure in the data, then the sources are not identiﬁable essentially (Hyv¨arinen & Pajunen, 1999).
Revised: in short), i.e. there is no temporal or similar structure in the data, then the sources are not identiﬁable essentially (Hyv¨arinen & Pajunen, 1999).

Original: This motivates many works to involve some structures to data for identiﬁability guarantee.
Revised: This motivates many works to involve some structures to data for identiﬁability guarantee.

Original: To deal with the unidentiﬁability problem, most existing works of nonliear ICA involve structures to data by restricting the sources in some plausible ways.
Revised: To deal with the unidentiﬁability problem, most existing works of nonliear ICA involve structures to data by restricting the sources in some plausible ways.

Original: One popular solution is to assume the sources have some kinds of temporal structures (Hyvarinen & Morioka, 2016; 2017).
Revised: One popular solution is to assume the sources have some kinds of temporal structures (Hyvarinen & Morioka, 2016; 2017).

Original: These works almost have no restriction on the mixing functions, but limit  data in the form of time series.
Revised: These works almost have no restriction on the mixing functions, but limit the data in the form of time series.

Original: Recent works (Hyvarinen et al., 2019; Khemakhem et al., 2020) extend the applicability of nonlinear ICA by involving auxiliary variables like labels of classes, and assume that the priors of sources are members of exponential family.
Revised: Recent works (Hyvarinen et al., 2019; Khemakhem et al., 2020) extend the applicability of nonlinear ICA by involving auxiliary variables like labels of classes, and assume that the priors of sources are members of exponential family.

Original: The theory of these work requires many classes in a dataset and each class should not be isolated in data space (Sorrenson et al., 2020), which is difﬁcult to be satisﬁed.
Revised: However, their works require many classes in a dataset and each class should not be isolated in data space (Sorrenson et al., 2020), which is difﬁcult to be satisﬁed.

Original: To see this, note that most datasets (e.g. CIFAR-10 (Krizhevsky et al., 2009)) just have very few classes, and usually each class is isolated in data space.
Revised: To see this, note that most datasets (e.g. CIFAR-10 (Krizhevsky et al., 2009)) just have very few classes, and usually each class is isolated in data space.

Original: Another direction of involving structures to data is to restrict the mixing function.
Revised: Another direction of involving structures to data is to restrict the mixing function.

Original: One way is to assume that the mixing function is post-nonlinear (Taleb & Jutten, 1999), i.e. each data variable is a nonlinear function of a linear combination of the sources.
Revised: One way is to assume that the mixing function is post-nonlinear (Taleb & Jutten, 1999), i.e. each data variable is a nonlinear function of a linear combination of the sources.

Original: This is obviously overly restrictive, and hence is impractical.
Revised: This is obviously overly restrictive, and hence is impractical.

Original: Some works provide a few heuristic settings on learning algorithms to facilitate identiﬁability (Watters et al., 2019; Sorrenson et al., 2020), e.g. Sorrenson et al.
Revised: Some works provide a few heuristic settings on learning algorithms to facilitate identiﬁability (Watters et al., 2019; Sorrenson et al., 2020), e.g. Sorrenson et al.

Original: (2020) empirically report that volume preservation of ﬂow-based models promotes disentanglement.
Revised: (2020) empirically report that volume preservation of ﬂow-based models promotes disentanglement.

Original: In this work, we explore a balancing direction: involve some natural structures to the mixing function, and meanwhile loosen the restrictions on the sources.
Revised: In this work, we explore a balancing direction: involve some natural structures to the mixing function, and meanwhile loosen the restrictions on the sources.

Original: Speciﬁcally, motivated by the empirical evidences by (Sorrenson et al., 2020), we assume the mixing function to be volume-preserving, and seek some natural restrictions on the sources for identiﬁability guarantee.
Revised: Speciﬁcally, motivated by the empirical evidences by (Sorrenson et al., 2020), we assume the mixing function to be volume-preserving (and a weaker condition is provided in Appendix B), and seek some natural restrictions on the sources for identiﬁability guarantee.

Original: With mild conditions, we establish novel and fundamental identiﬁability theorems, which mainly show that if there exist two different classes in the datasets, the true sources are identiﬁable.
Revised: With mild conditions, we establish novel and fundamental identiﬁability theorems, which mainly show that if there exist two distinct classes in the datasets, the true sources are identiﬁable.

Original: The proofs are essentially different to existing works, and provide some insights for nonlinear ICA: the main indeterminacy of a nonlinear ICA framework using volume-preserving transformations is the rotation of latent variables.
Revised: The proofs are essentially different to existing works, and provide some insights for nonlinear ICA: the main indeterminacy of a nonlinear ICA framework using volume-preserving mixing functions is the rotation of latent variables.

Original: Based on our identiﬁability theorems, we implement the proposed nonlinear ICA framework using a volume-preserving Flow-based model (Dinh et al., 2014), which requires merely two classes of data according to our theory.
Revised: Based on our identiﬁability theorems, we implement the proposed nonlinear ICA framework using a volume-preserving ﬂow-based model (Dinh et al., 2014), which requires merely two classes of data according to our theory.

Original: First, we  showin artiﬁcial data that even if some conditions in our identiﬁability theorems are not satisﬁed, our method still successfully identiﬁes the sources, which implies that there might exist some extensions of our theorems with weaker conditions.
Revised: First, we empirically show that with the most essential conditions on sources (two distinct classes with non-zero overlap) and mixing functions, the sources can be well identiﬁed, which implies that there might exist some extensions of our theorems with weaker conditions.

Original: Then we compare our implementation with the state-of-the-art identiﬁable model
Revised: Then we compare our implementation with the state-of-the-art identiﬁable model

Original: iVAE (Khemakhem et al., 2020) on artiﬁcialdataset and synthetic image dataset, and show that with volume preservation, ﬂow-based model achieves remarkably better results in terms of mean correlation coefﬁcient.
Revised: iVAE (Khemakhem et al., 2020) on artiﬁcial and synthetic image datasets, and show that our framework remarkably better results in terms of mean correlation coefﬁcient.

Original: More importantly, we  show that besides the given two classes, involving more classes cannot improve the performance of the volume-preserving ﬂow-based model.
Revised: More importantly, we empirically show that besides the given two classes, involving more classes cannot improve identiﬁability.

Original: Moreover, experiments on MNIST (LeCun et al., 1998)  indicate that the implementation is able to disentangle interpretable features using only two classes of data, which demonstrates the applicability of our theory.
Revised: Moreover, experiments on MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) indicate that the implementation is able to disentangle interpretable features using only two classes of data, which demonstrates the applicability of our theory.

Original: Our contributions are as follows
Revised: Our contributions are as follows:

Original: : 1) We present a new solution to the unidentiﬁability problem of nonlinear ICA by using the general volume-preserving transformation, as widely used in Flow-based models.
Revised: 1) We present a new solution to the unidentiﬁability problem of nonlinear ICA by using the general volume-preserving transformation, as widely used in ﬂow-based models.

Original: Speciﬁcally, our theoretical results lessen the requirement by existing works on the sources, at the slight cost of adding some natural conditions on the mixing function.
Revised: Speciﬁcally, our theoretical results lessen the requirement by existing works on the sources by reducing the number of required classes to 2 , at the slight cost of adding some natural conditions on the mixing function.

Original: Accordingly, all the restrictions are moderate, which facilitates the applicability of nonlinear ICA.
Revised: Accordingly , all the restrictions are moderate which facilitates the applicability of nonlinear ICA

Original: We establish two novel identiﬁability theorems.
Revised: We establish two novel identiﬁability theorems.

Original: Speciﬁcally, by using the friendly and general volume-preserving property, they guarantee the sources can be identiﬁed up to a point-wise nonlinearity and a point-wise linearity, respectively.
Revised: Speciﬁcally, by using the friendly and general volume-preserving property, they guarantee the sources can be identiﬁed up to a point-wise nonlinearity and a point-wise linearity, respectively.

Original: The proofs suggest that the main indeterminacy of a nonlinear ICA framework using volume-preserving is simply the rotation of latent variables.
Revised: The proofs suggest that the main indeterminacy of a nonlinear ICA framework with some natural conditions on mixing functions (volume-preservation or a weaker condition, see Appendix B) is simply the rotation of latent variables.

Original: The hope is to lay the foundation for a new path to identiﬁability.
Revised: The hope is to lay the foundation for a new path to identiﬁability.

Original: We conduct experiment on both synthetic and real-world data to verify our theory, and point out that even if some conditions in our theory arenot satisﬁed, our framework can successfully identify the true sources.
Revised: We conduct experiment on both synthetic and real-world data to verify our theory, and point out that as long as the most essential conditions on sources (two distinct classes with non-zero overlap) and mixing functions are satisﬁed, our framework can well identify the true sources.

Original: This means that our theory can be further extended.
Revised: This means that our theory can be further extended.

Original: Moreover,algorithm based on our framework remarkably excels the state-of-the-art nonlinear ICA method, which indicates that our framework is more powerful to identify the true sources.
Revised: Moreover, our framework remarkably excels the state-of-the-art nonlinear ICA method, which indicates that our framework is more powerful to identify the true sources.

Original:  B ACKGROUNDS
Revised:  B ACKGROUNDS

Original: In this section, we give some background on nonlinear ICA theory.
Revised: In this section, we provide some backgrounds on nonlinear ICA theory.

Original: We ﬁrst give a general deﬁnition of nonlinear ICA, and then explain its central problem, namely unidentiﬁability.
Revised: We ﬁrst give a general deﬁnition of nonlinear ICA, and then explain its central problem, namely unidentiﬁability.

Original: Based on this, we discuss some recently proposed solutions, including their advantages and disadvantages.
Revised: Based on this, we discuss some recent proposed solutions, including their advantages and disadvantages.

Original: 2. D EFINITION AND THE K EY P ROBLEM OF N ONLINEAR ICA
Revised: 2. D EFINITION AND THE K EY P ROBLEM OF N ONLINEAR ICA

Original: Consider an observed random vector x ∈ R d , we assume it is generated by an invertible nonlinear transformation f (called mixing function) using n independent latent variables s
Revised: Consider an observed random vector x ∈ R d , we assume it is generated by an invertible nonlinear transformation f (called mixing function) using n independent latent variables s

Original: = ( s 1 , · · · , s n ) (called independent components or sources) as
Revised: = ( s 1 , · · · , s n ) (called independent components or sources) as

Original: [Equation]
Revised: [Equation]

Original: where n ≤ d (Khemakhem et al., 2020).
Revised: where n ≤ d (Khemakhem et al., 2020).

Original: In earlier ICA theory it is commonly assumed that n = d , i.e. the number of independent components should be the same as the number of observed variables.
Revised: In earlier ICA theory it is commonly assumed that n = d , i.e. the number of independent components should be the same as the number of observed variables.

Original: As it is commonly believed that n (cid:28) d for real world data (Cayton, 2005; Narayanan & Mitter, 2010; Rifai et al., 2011), the deﬁnition above is much more practical.
Revised: As it is commonly believed that n (cid:28) d for real world data (Cayton, 2005; Narayanan & Mitter, 2010; Rifai et al., 2011), the deﬁnition above is much more practical.

Original: The goal of nonlinear ICA theory is to identify (or recover) the independent components s i based on the observations of x using a estimating function g .
Revised: The goal of nonlinear ICA theory is to identify (or recover) the independent components s i based on the observations of x using a estimating function g .

Original: Given the data distribution p ( x ) , if there exists a transformation g such that for arbitrary f , g ◦ f is a point-wise linear transformation, then the mixing model (Eq. 1) is said to be identiﬁable.
Revised: Given the data distribution p ( x ) , if there exists a transformation g such that for arbitrary f , g ◦ f is a point-wise linear transformation, then the mixing model (Eq. 1) is said to be identiﬁable.

Original: Unfortunately, if there are no more restrictions on the mixing model (Eq. 1), it is seriously unidentiﬁable (Hyv¨arinen & Pajunen, 1999; Locatello et al., 2019).
Revised: Unfortunately, if there are no more restrictions on the mixing model (Eq. 1), it is seriously unidentiﬁable (Hyv¨arinen & Pajunen, 1999; Locatello et al., 2019).

Original: Speciﬁcally, even if the components of the estimated latent variables z ≡ g ◦ f ( s ) are independent, z i can be the mixing of { s i } ni =1 .
Revised: Speciﬁcally, even if the components of the estimated latent variables z ≡ g ◦ f ( s ) are independent, z i can be the mixing of { s i } ni =1 .

Original: In other words, independence of components does not guarantee identiﬁability of the mixing model.
Revised: In other words, independence of components does not guarantee identiﬁability of the mixing model.

Original: E.g. suppose the prior p ( s ) is a factorial multivariate Gaussian, take a point-wise scaling on s such that p ( s ) becomes an isotropic multivariate Gaussian, then take a rotation around the center, the independence of components is unchanged, but the obtained components are the mixing of the original sources.
Revised: E.g. suppose the prior p ( s ) is a factorial multivariate Gaussian, take a point-wise scaling on s such that p ( s ) becomes an isotropic multivariate Gaussian, then take a rotation around the center, the independence of components is unchanged, but the obtained components are the mixing of the original sources.

Original: As arbitrary density functions can be transformed to a Gaussian (Locatello et al., 2019), this example shows that for any prior, the independence of components is insufﬁcient for identiﬁability.
Revised: As arbitrary density functions can be transformed to a Gaussian (Locatello et al., 2019), this example shows that for any prior, the independence of components is insufﬁcient for identiﬁability.

Original: 2. E XISTING S OLUTIONS FOR U NIDENTIFIABILITY P ROBLEM
Revised: 2. E XISTING S OLUTIONS FOR U NIDENTIFIABILITY P ROBLEM

Original: To obtain identiﬁability, it is necessary to restrict the mixing model.
Revised: To obtain identiﬁability, it is necessary to restrict the mixing model.

Original: There are two possible ways to involve restrictions: 1) restrict the sources s ; 2) restrict the mixing function f .
Revised: There are two possible ways to involve restrictions: 1) restrict the sources s ; 2) restrict the mixing function f .

Original: In the ﬁrst direction, one idea is to involve temporal structure to the sources s , i.e. assuming the sources are time series s ( t ) with time index t (Harmeling et al., 2003; Sprekeler et al., 2014; Hyvarinen & Morioka, 2016; 2017).
Revised: In the ﬁrst direction, one idea is to involve temporal structure to the sources s , i.e. assuming the sources are time series s ( t ) with time index t (Harmeling et al., 2003; Sprekeler et al., 2014; Hyvarinen & Morioka, 2016; 2017).

Original: This solution guarantees identiﬁability of the mixing model with few restrictions on the mixing function, but is limited in the setting of time series.
Revised: This solution guarantees identiﬁability of the mixing model with few restrictions on the mixing function, but is limited in the setting of time series.

Original: A more general way is to involve an auxiliary variable u and assume that conditioned on them, the prior of sources s is a factorial member of exponential family (Hyvarinen et al., 2019; Khemakhem et al., 2020):
Revised: A more general way is to involve an auxiliary variable u and assume that conditioned on them, the prior of sources s is a factorial member of exponential family (Hyvarinen et al., 2019; Khemakhem et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: where T i,j , λ i,j , Z i are the so-called sufﬁcient statistics, coefﬁcients and normalizing constant, respectively.
Revised: where T i,j , λ i,j , Z i are the so-called sufﬁcient statistics, coefﬁcients and normalizing constant, respectively.

Original: Q i is the so-called base measure, and is simply set to be 1 in many cases.
Revised: Q i is the so-called base measure, and is simply set to be 1 in many cases.

Original: k is the order of the distribution, and distributions with higher k are more ﬂexible.
Revised: k is the order of the distribution, and distributions with higher k are more ﬂexible.

Original: The auxiliary variable u is additionally observed, which can be time index, class label, etc.
Revised: The auxiliary variable u is additionally observed, which can be time index, class label, etc.

Original: This solution seems general, but it requires nk + 1 distinct values of u to guarantee identiﬁability, which is impossible in most datasets.
Revised: This solution seems general, but it requires nk + 1 distinct values of u to guarantee identiﬁability, which is impossible in most datasets.

Original: For example, if we assume that in MNIST (LeCun et al., 1998), the prior of s is a factorial multivarite Gaussian ( k = 2 ), and attempt to identify the sources using the 10 labels of the dataset based on the theory above, then we can identify 4 sources at most.
Revised: For example, if we assume that in MNIST (LeCun et al., 1998), the prior of s is a factorial multivarite Gaussian ( k = 2 ), and attempt to identify the sources using the 10 labels of the dataset based on the theory above, then we can identify 4 sources at most.

Original: However, the intrinsic dimensions is obviously far more than 4 (Pope et al., 2021), and hence we cannot identify the true sources.
Revised: However, the intrinsic dimensions is obviously far more than 4 (Pope et al., 2021), and hence we cannot identify the true sources.

Original: In the second direction, there are very few works to obtain identiﬁability by restricting the mixing function f in nonlinear ICA theory.
Revised: In the second direction, there are very few works to obtain identiﬁability by restricting the mixing function f in nonlinear ICA theory.

Original: One solution is to assume f to be post-nonlinear, i.e. { x j } dj =1 are nonlinear functions of the linear combination of sources, then the mixing model is identiﬁable (Taleb & Jutten, 1999).
Revised: One solution is to assume f to be post-nonlinear, i.e. { x j } dj =1 are nonlinear functions of the linear combination of sources, then the mixing model is identiﬁable (Taleb & Jutten, 1999).

Original: This condition is too restrictive (Hyvarinen et al., 2019) in practice.
Revised: This condition is too restrictive (Hyvarinen et al., 2019) in practice.

Original:  T HE P ROPOSED ICA F RAMEWORK
Revised:  T HE P ROPOSED ICA F RAMEWORK

Original: In this section, we propose our general framework for nonlinear ICA (see Fig. 1).
Revised: In this section, we propose our general framework for nonlinear ICA (see Fig. 1).

Original: 3. D EFINITION OF G ENERATIVE M ODEL
Revised: 3. D EFINITION OF G ENERATIVE M ODEL

Original: We assume that the sources s are independent conditioned on an auxiliary variable u : 
Revised: We assume that the sources s are independent conditioned on an auxiliary variable u : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p i is the i -th marginal distribution of the joint distribution p .
Revised: where p i is the i -th marginal distribution of the joint distribution p .

Original: Note that so far we do not restrict the density function of each source p
Revised: Note that so far we do not restrict the density function of each source p

Original: i
Revised: i

Original: ( s i | u ) to any particular functional form.
Revised: ( s i | u ) to any particular functional form.

Original: Hence our restrictions on the prior are looser than existing works (Hyvarinen et al., 2019; Khemakhem et al., 2020).
Revised: Hence our restrictions on the prior are looser than existing works (Hyvarinen et al., 2019; Khemakhem et al., 2020).

Original: estimated latent variables z is required a factorial multivariate Gaussian

Original: Next we introduce some coherent and basic restrictions on the mixing function f .
Revised: Next we introduce some coherent and basic restrictions on the mixing function f .

Original: First, we assume it is a homeomorphism 1 , mapping sources s to a vector of observed data variables x on a n -dimensional Riemannian manifold M embedded in R d :
Revised: First, we assume it is a homeomorphism 1 , mapping sources s to a vector of observed data variables x on a n -dimensional Riemannian manifold M embedded in R d :

Original: [Equation]
Revised: [Equation]

Original: This restriction is also natural, as it is believed that real world data often lies on a low dimensional manifold in high dimensional space (Rifai et al., 2011; Pope et al., 2021).
Revised: This restriction is also natural, as it is believed that real world data often lies on a low dimensional manifold in high dimensional space (Rifai et al., 2011; Pope et al., 2021).

Original: We further assume it is volume-preserving (Dinh et al., 2014), i.e. the volume of each inﬁnitesimal area on R n equals to that of the area on M obtained by this transformation.
Revised: We further assume it is volume-preserving (Dinh et al., 2014), i.e. the volume of each inﬁnitesimal area on R n equals to that of the area on M obtained by this transformation.

Original: The volume-preserving property is:
Revised: The volume-preserving property is:

Original: [Equation]
Revised: [Equation]

Original: ) where J f is the Jacobian matrix of the mixing function f from the view of differential manifold 2 .
Revised: where J f is the Jacobian matrix of the mixing function f from the view of differential manifold 2 .

Revised: Besides volume-preserving transformations, there exists a broader class of non-volume-preserving mixing functions to guarantee identiﬁability.

Revised: We introduce such class of functions in Appendix B.

Original: 3. D EFINITION OF E STIMATING M ODEL
Revised: 3. D EFINITION OF E STIMATING M ODEL

Original: Given the generative model above, next we seek an estimating model to identify the sources s from x .
Revised: Given the generative model above, next we seek an estimating model to identify the sources s from x .

Original: First, we set the estimating function g as a homeomorphism mapping the manifold M to R n :[Equation]
Revised: First, we set the estimating function g as a homeomorphism mapping the manifold M to R n :

Original: [Equation]
Revised: [Equation]

Original: where z are estimated latent variables.
Revised: where z are estimated latent variables.

Original: Note the estimating function g is set to be volume-preserving:
Revised: Note the estimating function g is set to be volume-preserving:

Original: [Equation]
Revised: [Equation]

Original: To identify s by z , naturally z should also be conditional independent.
Revised: To identify s by z , naturally z should also be conditional independent.

Original: Besides, we set the distribution of z conditioned on u as a factorial multivariate Gaussian:
Revised: Besides, we set the distribution of z conditioned on u as a factorial multivariate Gaussian:

Original: [Equation]
Revised: [Equation]

Original: This restriction on z  is actually an implicit one on sdue to the volume-preserving property of f ◦ g .
Revised: Actually, the model above gives a constraint to the priors p ( s | u ) . Speciﬁcally, q ( z | u ) must be the push-forward of p ( s | u ) through g ◦ f . As a result, the restriction on z above is actually an implicit one on s .

Original: To see this, note that a volume-preserving transformation does not change the volume of the distribution’s support.
Revised: To see this, note that a volume-preserving transformation does not change the volume of a distribution’s support.

Original: Hence, a multivariate uniform distribution cannot be transformed into a multivariate Gaussian by a volume-preserving function, because uniform distribution has limited support while the support of Gaussian is the whole space.
Revised: Hence, a multivariate uniform distribution cannot be transformed into a multivariate Gaussian by a volume-preserving function, because uniform distribution has limited support while the support of Gaussian is the whole space R n .

Original: Thus the prior of sources p ( s | u ) cannot be a multivariate uniform distribution in our theory, and at least its support should coverthe whole space R n . 
Revised: Thus the priors p ( s | u ) cannot be a multivariate uniform distribution in our theory, and at least its support should cover R n . [Figure]

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: It is obvious that a generative model deﬁned by Eq.
Revised: It is obvious that a generative model deﬁned by Eq.

Original: 3-Eq.
Revised: 3-Eq.

Original: 5 is still unidentiﬁable using an estimating model deﬁned by Eq.
Revised: 5 is still unidentiﬁable using an estimating model deﬁned by Eq.

Original: 6-Eq. 8.
Revised: 6-Eq. 8.

Original: For example, if f is a point-wise nonlinear transformation mapping p ( s | u ) to an isotropic multivariate Gaussian, and g is a rotation around the center with arbitrary angle, then the conditions Eq.
Revised: For example, if f is a point-wise nonlinear transformation mapping p ( s | u ) to an isotropic multivariate Gaussian, and g is a rotation around the center with arbitrary angle, then the conditions Eq.

Original: 3-Eq.
Revised: 3-Eq.

Original: 8 are satisﬁed, but the obtained z i is still the mixing of s .
Revised: 8 are satisﬁed, but the obtained z i is still the mixing of s .

Original: Therefore, we further seek some mild restrictions to guarantee the identiﬁability of the generative model.
Revised: Therefore, we further seek some mild restrictions to guarantee the identiﬁability of the generative model.

Original: The goal is to prove that under some mild restrictions, the estimated latent variables z are the recoveries of the sources s up to a point-wise linear transformation, i.e. g ◦ f is point-wise linear function.
Revised: The goal is to prove that under some mild restrictions, the estimated latent variables z are the recoveries of the sources s up to a point-wise linear transformation, i.e. g ◦ f is point-wise linear function.

Original: Our main result is the following theorems:is
Revised: Our main result is the following theorems:

Original: and M [Figure] Theorem 1 ( nonlinear identiﬁability ) Assume data points are sampled from a model deﬁned by
Revised: Theorem 1 ( nonlinear identiﬁability ) Assume data points are sampled from a model deﬁned by

Original: to
Revised: to

Original: Eq. 3-Eq. 8, and there exist two distinct observations of u , denoted by u (1) and u(2) , such that:
Revised: Eq. 3-Eq. 8, and there exist two distinct observations of u , denoted by u 1) and u such that

Original:  (i) Both f and g have all second order derivatives.
Revised: : (i) Both f and g have all second order derivatives.

Original: (ii) µ
Revised: (ii) µ

Original: i ( u (1) )
Revised: i ( u (1) )

Original: = µ i ( u (2) ) , ∀ i ∈ [ n ] .
Revised: = µ i ( u (2) ) , ∀ i ∈ [ n ] .

Original: (iii) (cid:110) σ 1 ( u (2) )
Revised: (iii) (cid:110) σ 1 ( u (2) )

Original: σ 1 ( u (1) ) , · · · , σ n ( u (2) )
Revised: σ 1 ( u (1) ) , · · · , σ n ( u (2) )

Original: σ n ( u (1) ) (cid:111) are distinct.
Revised: σ n ( u (1) ) (cid:111) are distinct.

Original: Then g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.
Revised: Then g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.

Original: Remarks.
Revised: Remarks.

Original: The theorem (proof is given in Appendix A) guarantees that the sources s of the generative model above are identiﬁable up to a nonlinear point-wise transformation using the estimating model above, and hence the generative model is identiﬁable.
Revised: The theorem (proof is given in Appendix A) guarantees that the sources s of the generative model above are identiﬁable up to a nonlinear point-wise transformation using the estimating model above, and hence the generative model is identiﬁable.

Original: In the following, we discuss the meaning of the conditions (ii) and (iii) above, and omit condition (i) since it is very mild and trivial.
Revised: In the following, we discuss the meaning of the conditions (ii) and (iii) above, and omit condition (i) since it is very mild and trivial.

Original: Condition (ii) means that the two multivariate Gaussians q ( z | u (1) ) and q ( z | u (2) ) are concentric, which guarantees that samples from the two Gaussians have large enough “overlap”.
Revised: Condition (ii) means that the two multivariate Gaussians q ( z | u (1) ) and q ( z | u (2) ) are concentric, which guarantees that samples from the two Gaussians have large enough “overlap”.

Original: Otherwise, if the centers of the two Gaussians have large distance, samples from these two Gaussians are likely to be absolutely disjoint when using Monte Carlo sampling.
Revised: Otherwise, if the centers of the two Gaussians have large distance, samples from these two Gaussians are likely to be absolutely disjoint when using Monte Carlo sampling.

Original: In this case, for the estimating function g there is equivalently one Gaussian because it can deal with samples from the two Gaussians separately, and hence identiﬁability is not guaranteed.
Revised: In this case, for the estimating function g there is equivalently one Gaussian because it can deal with samples from the two Gaussians separately, and hence identiﬁability is not guaranteed.

Original: In our proof (see Appendix A), however, it seems that this condition is not theoretically necessary, as the supports of the two Gaussians are the whole space and hence they theoretically have overlap everywhere.
Revised: In our proof (see Appendix A), however, it seems that this condition is not theoretically necessary, as the supports of the two Gaussians are the whole space and hence they theoretically have overlap everywhere.

Original: How to prove an identiﬁability theorem without this condition is very important, and we leave this problem to future works.
Revised: How to prove an identiﬁability theorem without this condition is very important, and we leave this problem to future works.

Original: Condition (iii) means that there do not exist two estimated latent variables such that their variances of the two Gaussians are proportional.
Revised: Condition (iii) means that there do not exist two estimated latent variables such that their variances of the two Gaussians are proportional.

Original: If this condition is not fulﬁlled, then we can easily construct another latent variables which are mixing of the estimated latent variables, and hence the identiﬁability is impossibly guaranteed.
Revised: If this condition is not fulﬁlled, then we can easily construct another latent variables which are mixing of the estimated latent variables, and hence the identiﬁability is impossibly guaranteed.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, assume σ 1 ( u (2) )
Revised: 2, assume σ 1 ( u (2) )

Original: σ 1 ( u (1) )
Revised: σ 1 ( u (1) )

Original: =
Revised: =

Original: σ 2 ( u (2) )
Revised: σ 2 ( u (2) )

Original: σ 2 ( u (1) ) , let ( z (cid:48) 1 , z (cid:48) 2 ) =
Revised: σ 2 ( u (1) ) , let ( z (cid:48) 1 , z (cid:48) 2 ) =

Original: ( z 1 , z 2 ) SR , where S = diag
Revised: ( z 1 , z 2 ) SR , where S = diag

Original: [Equation]
Revised: [Equation]

Original: is a scaling matrix and R is an arbitrary rotation matrix of order 2 , hence SR is a volume-preserving transformation.
Revised: is a scaling matrix and R is an arbitrary rotation matrix of order 2 , hence SR is a volume-preserving transformation.

Original: Then ( z (cid:48) 1 , z (cid:48) 2 ) are independent conditioned on both u (1) and u (2) , and hence they are also a set of estimated latent variables, but obviously they are the mixing of ( z 1 , z 2 ) .
Revised: Then ( z (cid:48) 1 , z (cid:48) 2 ) are independent conditioned on both u (1) and u (2) , and hence they are also a set of estimated latent variables, but obviously they are the mixing of ( z 1 , z 2 ) .

Original: Therefore, condition (iii) is a natural and necessary restriction for identiﬁability.
Revised: Therefore, condition (iii) is a natural and necessary restriction for identiﬁability.

Original: Although (ii) and (iii) are conditions on the distributions of the estimated latent variables z , they are also implicit restrictions on the prior of sources s .
Revised: Although (ii) and (iii) are conditions on the distributions of the estimated latent variables z , they are also implicit restrictions on the prior of sources s .

Original: To see this, note that
Revised: To see this, note that

Original: z = g ◦ f ( s ) and g ◦ f is a function constrained by Eq.
Revised: z = g ◦ f ( s ) and g ◦ f is a function constrained by Eq.

Original: 4-Eq. 6, and hence (ii) and (iii) implicitly restrict the prior through g ◦ f .
Revised: 4-Eq. 6, and hence (ii) and (iii) implicitly restrict the prior through g ◦ f .

Original: If we further restrict the prior p ( s | u ) to a multivariate Gaussian, then we can reduce nonlinear indeterminacy to linear indeterminacy:
Revised: If we further restrict the prior p ( s | u ) to a multivariate Gaussian, then we can reduce nonlinear indeterminacy to linear indeterminacy:

Original: Theorem 2 ( linear identiﬁability ) Assume the hypotheses of Theorem 1 hold, and p ( s | u (1) ) and p ( s | u (2) ) are multivariate Gaussians.
Revised: Theorem 2 ( linear identiﬁability ) Assume the hypotheses of Theorem 1 hold, and p ( s | u (1) ) and p ( s | u (2) ) are multivariate Gaussians.

Original: Then g ◦ f is a composition of a point-wise linear transformation and a permutation.
Revised: Then g ◦ f is a composition of a point-wise linear transformation and a permutation.

Original: In this case, the sources will have similar restrictions with the estimated latent variables.
Revised: In this case, the sources will have similar restrictions with the estimated latent variables.

Original: Specifically, let µ s ( u ) and σ s ( u ) be the mean and variance of p ( s | u ) , then we have: (ii’) µ si ( u (1) )
Revised: Specifically, let µ s ( u ) and σ s ( u ) be the mean and variance of p ( s | u ) , then we have: (ii’) µ si ( u (1) )

Original: = µ si ( u (2) ) ,
Revised: = µ si ( u (2) ) ,

Original: ∀ i ∈ [ n ] ; and (iii’)
Revised: ∀ i ∈ [ n ] ; and (iii’)

Original: (cid:110) σ s 1 ( u (2) )
Revised: (cid:110) σ s 1 ( u (2) )

Original: σ s 1 ( u (1) ) , · · · , σ sn ( u (2) )
Revised: σ s 1 ( u (1) ) , · · · , σ sn ( u (2) )

Original: σ sn ( u (1) )
Revised: σ sn ( u (1) )

Original: (cid:111) are distinct.
Revised: (cid:111) are distinct.

Original: The proofs of the theorems above (see Appendix A) give us some key insights for nonlinear ICA.
Revised: The proofs of the theorems above (see Appendix A) give us some key insights for nonlinear ICA.

Original: Speciﬁcally, a nonlinear ICA framework using volume-preserving transformations only has three kinds of indeterminacy: the point-wise nonlinear transformation, the permutation and the rotation of the latent variables.
Revised: Speciﬁcally, a nonlinear ICA framework using volume-preserving transformations only has three kinds of indeterminacy: the point-wise nonlinear transformation, the permutation and the rotation of the latent variables .

Original: Therefore, resolving the indeterminacy of rotation by condition (iii) will lead to identiﬁability, and the indeterminacy of point-wise nonlinear transformation can be reduced to point-wise linear transformation by the Gaussianity of p ( s | u ) .
Revised: Therefore, resolving the indeterminacy of rotation by condition (iii) will lead to identiﬁability, and the indeterminacy of point-wise nonlinear transformation can be reduced to point-wise linear transformation by the Gaussianity of p ( s | u ) .

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: As our theory requires an invertible estimating function, we set it as a volume-preserving Flowbased model (Dinh et al., 2014).
Revised: As our theory requires an invertible estimating function, we set it as a volume-preserving Flowbased model (Dinh et al., 2014).

Original: The estimating function is denoted by g ( · ; θ ) , where θ refers to model parameters.
Revised: The estimating function is denoted by g ( · ; θ ) , where θ refers to model parameters.

Original: Given the data variables x , g ( x ; θ ) denotes the estimated latent variables z .
Revised: Given the data variables x , g ( x ; θ ) denotes the estimated latent variables z .

Original: Given a dataset D = (cid:8) ( x (1 , 1) , u (1) ) , · · · , ( x (1 ,M ) , u (1) ) , ( x (2 , 1) , u (2) ) , · · · , ( x (2 ,N ) , u (2) ) (cid:9) , where there exist only two labels u (1) and u (2) , we can construct a loss function according to our theory.
Revised: Given a dataset D = (cid:8) ( x (1 , 1) , u (1) ) , · · · , ( x (1 ,M ) , u (1) ) , ( x (2 , 1) , u (2) ) , · · · , ( x (2 ,N ) , u (2) ) (cid:9) , where there exist only two labels u (1) and u (2) , we can construct a loss function according to our theory.

Original: As in our theory, the conditional distribution of the estimated latent variables should be a factorial multivariate Gaussian, we have to push (cid:8) g
Revised: As in our theory, the conditional distribution of the estimated latent variables should be a factorial multivariate Gaussian, we have to push (cid:8) g

Original: ( x (1 ,i ) ; θ ) (cid:9) M i
Revised: ( x (1 ,i ) ; θ ) (cid:9) M i

Original: =1 and (cid:8) g
Revised: =1 and (cid:8) g

Original: ( x (2 ,i ) ; θ ) (cid:9) N i =1 to q ( z | u (1) ) and q ( z | u (2) ) (deﬁned by Eq. 14) respectively.
Revised: ( x (2 ,i ) ; θ ) (cid:9) N i =1 to q ( z | u (1) ) and q ( z | u (2) ) (deﬁned by Eq. 14) respectively.

Original: Therefore, we minimize the negative log-likelihood following (Sorrenson et al., 2020):
Revised: Therefore, we minimize the negative log-likelihood following (Sorrenson et al., 2020):

Original: [Equation]
Revised: [Equation]

Revised: To optimize the loss above, ﬁrst the number of sources n should be given, but it is usually unknown.

Revised: Fortunately, we can deal with this problem by integrating dimension reduction into the loss, and hence the underlying dimensionality can be estimated by learning.

Original: Fortunately, this loss  can be further simpliﬁed as  (see Appendix C for details):
Revised: Speciﬁcally, the loss above can be further simpliﬁed as follows, which has the ability of dimension reduction (see Appendix C for detailed derivation and discussion):

Original: [Equation]
Revised: [Equation]

Original: where σ i ( u ( k ) ; θ ) = (cid:113) E ( x , u ( k ) ) ∈D (cid:2)
Revised: where σ i ( u ( k ) ; θ ) = (cid:113) E ( x , u ( k ) ) ∈D (cid:2)

Original: g i ( x ; θ ) − E ( x , u ( k ) ) ∈D
Revised: g i ( x ; θ ) − E ( x , u ( k ) ) ∈D

Original: [ g i ( x ; θ )] (cid:3) 2 .
Revised: [ g i ( x ; θ )] (cid:3) 2 .

Original: We utilize the simpliﬁed version throughout our experiments.
Revised: We utilize the simpliﬁed version throughout our experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Although our Theorem 1 does not explicitly restrict the particular function forms ofthe prior p ( s | u ) , it implicitly requires p ( s | u ) to be identical to a factorial multivariate Gaussian, up to volumepreserving transformations.
Revised: Although our Theorem 1 does not explicitly restrict the functional form of p ( s | u ) , it implicitly requires p ( s | u ) to be identical to a factorial multivariate Gaussian, up to volume-preserving transformations.

Original: However, it is difﬁcult to verify whether this implicit restriction is satisﬁed given a prior.
Revised: However, it is difﬁcult to verify whether this implicit restriction is satisﬁed given a dataset.

Original: Therefore, our experiments are mainly based on Theorem 2, in which the prior is also assumed to be a factorial multivariate Gaussian.
Revised: Therefore, our experiments are mainly based on Theorem 2, in which the prior is also assumed to be a factorial multivariate Gaussian.

Original: Hence, the main goal of this section is to show the performance of our framework to achieve linear identiﬁability in this case.
Revised: Hence, the main goal of this section is to show the performance of our framework to achieve linear identiﬁability in this case.

Original: 6. P ROTOCOL
Revised: 6. P ROTOCOL

Original: Datasets.
Revised: Datasets.

Original: We run experiments both on a artiﬁcial dataset and a synthesized image dataset called “Triangles”, as well as on real-world image dataset MNIST (LeCun et al., 1998).
Revised: We run experiments both on a artiﬁcial dataset and a synthetic image dataset called “Triangles”, as well as on real-world image datasets MNIST (LeCun et al., 1998) and CelebA

Original: To generate the the artiﬁcial data, following (Sorrenson et al., 2020), we synthesize the sources and observed data by two steps: i) First, two sources are sampled from a 2-dim mixture of two Gaussians with their mean both (0 . 0 , 0 . 0) , and with variances (1 . 0 , 0 . 5) and (0 . 5 , 1 . 0) , respectively. From each Gaussian, 5 , 000 points are sampled.
Revised: (Liu et al., 2015).

Original: Then an 8-dim standard Gaussian noise scaled by 0 .

Original: 01 is concatenated with them. ii) The observed data is generated from the 10-dim sources, by a randomly initialized GIN (Sorrenson et al., 2020) with 8 fully connected coupling blocks. To quantitatively test our method on image datasets, we have synthesized a gray-scale image dataset called “Triangles”.
Revised: The generative processes of the artiﬁcial dataset and the synthetic image dataset are described in Appendix D.

Original: Speciﬁcally, we choose to generate a grey-scale image dataset of 2-D shapes similar with dSprites (Matthey et al., 2017), in which all shapes are right triangles generated from factors: rotation, width (length of the horizontal edge), height (length of the vertical edge) and gray level.

Original: These factors are viewed as sources, and the observations of sources are sampled from a mixture of two Gaussians with random means and variances.

Original: More details can be found in Appendix D.

Original: Model speciﬁcation.
Revised: Model speciﬁcation.

Original: To implement the volume-preserving estimating function g ( · ; θ ) , we choose a volume-preserving ﬂow called GIN (Sorrenson et al., 2020), which is a volume-preserving version of
Revised: To implement the volume-preserving estimating function g ( · ; θ ) , we choose a volume-preserving ﬂow called GIN (Sorrenson et al., 2020), which is a volume-preserving version of

Original: RealNVP (Dinh et al., 2017).
Revised: RealNVP (Dinh et al., 2017).

Original: For experiments on artiﬁcial data, the network is constructed by fully connected coupling blocks.
Revised: For experiments on artiﬁcial data, the network is constructed by fully connected coupling blocks.

Original: For image datasets: Triangles and MNIST, the networks are constructed by both convolutional coupling blocks and fully connected coupling blocks.
Revised: For image datasets: Triangles and MNIST, the networks are constructed by both convolutional coupling blocks and fully connected coupling blocks.

Original: The parameters of each network are updated by minimizing the loss function in Eq.
Revised: The parameters of each network are updated by minimizing the loss function in Eq.

Original: 10 using an Adam optimizer (Kingma & Ba, 2014).
Revised: 10 using an Adam optimizer (Kingma & Ba, 2014).

Original: Details of the networks and their optimization refer to (Sorrenson et al., 2020).
Revised: Details of the networks and their optimization refer to (Sorrenson et al., 2020).

Original: Performance metric.
Revised: Performance metric.

Original: To quantitatively compare the performance of different methods, we compute the mean correlation coefﬁcient (MCC) (Khemakhem et al., 2020) between the sources and the estimated latent variables.
Revised: To quantitatively compare the performance of different methods, we compute the mean correlation coefﬁcient (MCC) (Khemakhem et al., 2020) between the sources and the estimated latent variables.

Original: The computation of MCC consists of three steps: i) calculate all pairs of correlation coefﬁcients between the sources and estimated latent variables; ii) solve a linear sum assignment problem to assign one estimated latent variable to each source for highest total correlation coefﬁcients; iii) take the average correlation coefﬁcients of the obtained pairs.
Revised: The computation of MCC consists of three steps: i) calculate all pairs of correlation coefﬁcients between the sources and estimated latent variables; ii) solve a linear sum assignment problem to assign one estimated latent variable to each source for highest total correlation coefﬁcients; iii) take the average correlation coefﬁcients of the obtained pairs.

Original: A high MCC means that we successfully identify the true sources, up to point-wise transformations.
Revised: A high MCC means that we successfully identify the true sources, up to point-wise transformations.

Original: 6. R ESULTS ON A RTIFICIAL D ATASET AND T RIANGLES
Revised: 6. R ESULTS ON A RTIFICIAL D ATASET AND T RIANGLES

Original: On artiﬁcial dataset and Triangles, the labels of Gaussians in the prior of sources are used as the observations of the auxiliary variabls u required in our theory.
Revised: On artiﬁcial dataset and Triangles, the labels of Gaussians in the prior of sources are used as the observations of the auxiliary variables u required in our theory.

Original: We mainly aim at verifying our theory, and comparing our framework with the state-of-the-art nolinear ICA method, namely iVAE (Khemakhem et al., 2020), which is based on variational autoencoders (VAE) (Kingma & Welling, 2013).
Revised: We mainly aim at verifying our theory, and comparing our framework with the state-of-the-art nonlinear ICA method, namely iVAE (Khemakhem et al., 2020), which is based on variational autoencoders (VAE) (Kingma & Welling, 2013).

Original: 6.2. A BLATION STUDY
Revised: 6.2. A BLATION STUDY

Original: [Table]
Revised: [Table]

Original: To verify whether the conditions in our theory are necessary for identifying the true sources, we perform an ablation study on artiﬁcial data.
Revised: To verify what conditions in our theory are essential for identifying the true sources, we perform an ablation study on artiﬁcial data.

Original: The chosen conditions include i) the concentric property of two Gaussians p ( s | u (1) ) and p ( s | u (1) ) , denoted by µ si ( u (1) )
Revised: The chosen conditions include i) the concentric property of two Gaussians p ( s | u (1) ) and p ( s | u (1) ) , denoted by µ si ( u (1) )

Original: = µ si ( u (2) ) ; ii) the variances of the two Gaussains should be not proportional, denoted by σ s 1 ( u (2) )
Revised: = µ si ( u (2) ) ; ii) the variances of the two Gaussains should be not proportional, denoted by σ s 1 ( u (2) )

Original: σ s 1 ( u (1) )
Revised: σ s 1 ( u (1) )

Original: (cid:54) =
Revised: (cid:54) =

Original: σ s 2 ( u (2) )
Revised: σ s 2 ( u (2) )

Original: σ s 2 ( u (1) ) ; iii) the the volume-preserving property of mixing function f ; iv) the existence of two distinct observations of auxiliary variable u (1) and u (2) ; v) the existence of overlap of samples from p ( s | u (1) ) and p ( s | u (1) ) .
Revised: σ s 2 ( u (1) ) ; iii) the the volume-preserving property of mixing function f ; iv) the existence of two distinct observations of auxiliary variable u (1) and u (2) ; v) the existence of overlap of samples from p ( s | u (1) ) and p ( s | u (1) ) .

Original: For verifying experiment of each condition, we synthesize an artiﬁcial dataset in which the condition is not satisﬁed.
Revised: For verifying experiment of each condition, we synthesize an artiﬁcial dataset in which the condition is not satisﬁed.

Original: The quantitative results of our framework when the conditions above are not satisﬁed are shown in
Revised: The quantitative results of our framework when the conditions above are not satisﬁed are shown in

Original: Table 1, and the qualitative results are shown in Appendix E for the sake of brevity.
Revised: Table 1, and the qualitative results are shown in Appendix E for the sake of brevity.

Original: The baseline is the case that all conditions are satisﬁed.

Original: Note that in the case of µ s 1 ( u (1) ) (cid:54) =
Revised: Note that in the case of µ s 1 ( u (1) ) (cid:54) =

Original: µ s 1 ( u (2) ) , samples from p ( s | u (1) ) and p ( s | u (2) ) have non-zero overlap (see Appendix E).
Revised: µ s 1 ( u (2) ) , samples from p ( s | u (1) ) and p ( s | u (2) ) have non-zero overlap.

Original: Over these experimental results we made the following observations
Revised: Over these experimental results, we made the following observations:

Original: : 1) As long as samples from p ( s | u (1) ) and p ( s | u (2) ) have non-zero overlap, GIN will almost perfectly identify the true sources, even if the p ( s | u (1) ) and p ( s | u (2) ) are not concentric.
Revised: 1) As long as samples from p ( s | u (1) ) and p ( s | u (2) ) have non-zero overlap, GIN will almost perfectly identify the true sources, even if the p ( s | u (1) ) and p ( s | u (2) ) are not concentric.

Original: However, when the overlap vanishes, the performance of GIN has a sharp decline.
Revised: However, when the overlap vanishes, the performance of GIN has a sharp decline.

Original: These results indicate that the condition (ii) in our Theorem 1 is not necessary.
Revised: These results indicate that the condition (ii) in our Theorem 1 is not necessary, but non-zero overlap is crucial for identiﬁability.

Original: 2) When σ s 1 ( u (2) ) σ s 1 ( u (1) )
Revised: When σ s 1 ( u (2) ) σ s 1 ( u (1) )

Original: (cid:54) =
Revised: (cid:54) =

Original: σ s 2 ( u (2) )
Revised: σ s 2 ( u (2) )

Original: σ s 2 ( u (1) ) or u (1) =
Revised: σ s 2 ( u (1) ) or u (1) =

Original: u (2) , the sources are unidentiﬁable according to our theoretical analysis, but the experimental resultshere show that in these cases the true sources still be well identiﬁed.
Revised: u (2) , the sources are unidentiﬁable according to our theoretical analysis, but the experimental results show that in these cases the true sources still be well identiﬁed.

Revised: This is not contradictory to the necessity of condition (iii) in Theorem 1 and two distinct classes, but indicates that there exist some biases in learning process to reduce the indeterminacy of rotation of latent variables.

Original: We conjecture that this isdue to the mini-batch sampling.
Revised: We conjecture that one bias is mini-batch sampling.

Original: As in mini-batchsampling, the empirical distribution of different mini-batch can be very different, while according to our theory, two different priors are almost sufﬁcient to guarantee identiﬁability using a volume-preserving transformation. Therefore, mini-batch sampling is an inductive bias for identiﬁability.
Revised: As in mini-batch

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Obviously, the fourth rows correspond to rotation, , respectively.
Revised: Obviously, the four rows correspond to rotation, width, height and gray level, respectively.

Original: Even when the mixing function f (not the estimating function g ) is non-volume-preserving, the true sources can be successfully identiﬁed using a volume-preserving estimating function. This suggests that besides volume-preserving transformations, there might exists a broader class of functions which still leads to identiﬁability in our theory.
Revised: sampling, the empirical distributions of different mini-batches can be very different, while according to our theory, two different priors are almost sufﬁcient to guarantee identiﬁability. 3) When the mixing function f (not the estimating function g ) is non-volume-preserving, the recovery of sources can be somewhat unsuccessful.

Original: We provide such a class of functions in Appendix B.
Revised: To see this, note that in Fig. 7(d), the recovery of ground truth has some visual distortions. This demonstrates that involving some restrictions to f (like volume-preservation or the condition introduced in Appendix B) is necessary.

Original: 6.2. Q UALITATIVE RESULTS
Revised: 6.2. Q UALITATIVE RESULTS

Original: Results on artiﬁcial data.
Revised: Results on artiﬁcial data.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, both iVAE and GIN can successfully identify the true sources from their nonlinear mixing data points.
Revised: 3, both iVAE and GIN can successfully identify the true sources from their nonlinear mixing data points.

Original: While GIN almost perfectly recovers the ground truth, yet the recovery by iVAE has some distortions everywhere.
Revised: While GIN almost perfectly recovers the ground truth, yet the recovery by iVAE has some distortions everywhere.

Original: Results on Triangles.
Revised: Results on Triangles.

Original: For qualitative comparison of iVAE and GIN, we plot traversals of the estimated latent variables by them, which are obtained by three steps: i) deal with the assignment problem of correlation coefﬁcients between 4 sources and all estimated latent variables, and choose the assigned latent variables for traversals, denoted by { z i } 4 i =1 ; ii) estimate the variances of the chosen latent variables { σ i } 4 i =1 ; iii) manipulate the value of the i -th latent variable between ( − 2 σ i , 2 σ i ) while keep other estimated latent variables unchanged, and then generate the i -th row of images.[Table]
Revised: For qualitative comparison of iVAE and GIN, we plot traversals of the estimated latent variables by them, which are obtained by three steps: i) deal with the assignment problem of correlation coefﬁcients between 4 sources and all estimated latent variables, and choose the assigned latent variables for traversals, denoted by { z i } 4 i =1 ; ii) estimate the variances of the chosen latent variables { σ i } 4 i =1 ; iii) manipulate the value of the i -th latent variable between ( − 2 σ i , 2 σ i ) while keep other estimated latent variables unchanged, and then generate the i -th row of images.

Original: In Fig.
Revised: In Fig.

Original: 4, the qualitative performances of iVAE and GIN are distinct.
Revised: 4, the qualitative performances of iVAE and GIN are distinct.

Original: Each plot estimated latent variable of iVAE is a mixing of the true sources, e.g. the 4 -th row of Fig. 4(a) is a mixing of rotation and color.
Revised: Each plot estimated latent variable of iVAE is a mixing of the true sources, e.g. the 4 -th row of Fig. 4(a) is a mixing of rotation and color.

Original: Compared with this, the plot estimated latent variables by GIN obviously correspond to the true sources: rotation, width, height and gray level, respectively.
Revised: Compared with this, the plot estimated latent variables by GIN obviously correspond to the true sources: rotation, width, height and gray level, respectively.

Original: This shows that our framework is much better than iVAE qualitatively, and is able to identify the true sources from images.
Revised: This shows that our framework is much better than iVAE qualitatively, and is able to identify the true sources from images.

Original: These results demonstrates that our framework is much more powerful than iVAE for identifying true sources on both artiﬁcial data and synthetic images with a few classes.
Revised: These results demonstrates that our framework is much more powerful than iVAE for identifying true sources on both artiﬁcial data and synthetic images with a few classes.

Original: [Figure]
Revised: 6.2. Q UANTITATIVE RESULTS .

Original: The quantitative results are shown in Table 2, in which ‘- m c’ means the model is trained and tested on a dataset with m classes.
Revised: The quantitative results are shown in Table 2, in which ‘- m c’ means the model is trained and tested on a dataset with m classes.

Original: According to the theory by (Hyvarinen et al., 2019; Khemakhem  et al., 2020), iVAE requires 9 classes on Triangles and 5 classes on our artiﬁcial data, hence we set iVAE9c, iVAE-5c and iVAE-2c for fair comparisions with GIN.
Revised: According to the theory by (Hyvarinen et al., 2019; Khemakhem [Figure] et al., 2020), iVAE requires 9 classes on Triangles and 5 classes on our artiﬁcial data, hence we set iVAE-9c, iVAE-5c and iVAE-2c for fair comparisions with GIN.

Original: As for GIN, we set GIN-9c,GIN-2c and
Revised: As for GIN, we set GIN-9c,

Original:  GIN-1c to verify whether 2 classes predicted by our theory are sufﬁcient and necessary.
Revised: GIN-2c and GIN-1c to verify whether 2 classes predicted by our theory are sufﬁcient and necessary.

Original:  As shown in Table 2, for identifying true sources, GIN excels iVAE w.r.t.
Revised: [Table] As shown in Table 2, for identifying true sources, GIN excels iVAE w.r.t.

Original: MCC on both artiﬁcial data and synthetic images.
Revised: MCC on both artiﬁcial data and synthetic images.

Original: This might be due to the volume-preserving essence of GIN, which provides a strong but natural restriction for identiﬁability according to our theory.
Revised: This might be due to the volume-preserving essence of GIN, which provides a strong but natural restriction for estimating functions according to our theory.

Original: Also, besides the given two classes, increasing the number of classes cannot markedly improve the performance of GIN, while reducing number of classes to 1 leads to a sharp decline of MCC.
Revised: Also, besides the given two classes, increasing the number of classes cannot markedly improve the performance of GIN, while reducing number of classes to 1 leads to a sharp decline of MCC.

Original: This means for GIN, two classes of data are sufﬁcient and necessary for identiﬁability.
Revised: This means for GIN, two classes of data are sufﬁcient and necessary for identiﬁability.

Original: These results are consistent with our theory.
Revised: These results are consistent with our theory.

Original: 6. R ESULTS ON R EAL W ORLD D ATASETS
Revised: 6. R ESULTS ON R EAL W ORLD D ATASETS

Original: To verify whether our framework is able to identify sources from real world datasets, we conduct experiments on MNIST.
Revised: To verify whether our framework is able to identify sources from real world datasets, we conduct experiments on MNIST and CelebA, and the setting and results on CelebA are reported in Appendix

Original: We pick images with digits ‘1’ and ‘7’ because they probably have nonzero overlap, and use their labels as observations of the auxiliary variable.
Revised: F. On MNIST, we pick images with digits ‘1’ and ‘7’ because they probably have non-zero overlap, and use their labels as observations of the auxiliary variable.

Original: As there is no ground truth in MNIST, we cannot compute the MCC score, and hence the qualitative results are mainly reported.
Revised: As there is no ground truth in MNIST, we cannot compute the MCC score, and hence the qualitative results are mainly reported.

Original: Moreover, the estimated latent variables with high standard deviations are chosen to plot, as such variables are probably more meaningful (Sorrenson et al., 2020).
Revised: Moreover, the estimated latent variables with high standard deviations are chosen to plot, as such variables are probably more meaningful (Sorrenson et al., 2020).

Original: We hope the estimated latent variables will correspond to some interpretable attributes in the dataset, which are viewed as true sources in disentanglement literatures (Bengio et al., 2013; Locatello et al., 2019).
Revised: We hope the estimated latent variables will correspond to some interpretable attributes in the dataset, which are viewed as true sources in disentanglement literatures (Bengio et al., 2013; Locatello et al., 2019).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, the estimated latent variables with top 4 standard  are highly interpretable, corresponding to digits’ rotation, bend, thickness and bend of the horizontal bar, respectively.
Revised: 5, the estimated latent variables with top 4 standard deviations are highly interpretable, corresponding to digits’ rotation, bend, thickness and bend of the horizontal bar, respectively.

Original: These results by our framework are comparable with fully utilizing the 10 classes of MNIST (Sorrenson et al., 2020).
Revised: These results by our framework are comparable with fully utilizing the 10 classes of MNIST (Sorrenson et al., 2020).

Original: Therefore, two classes are probably sufﬁcient for identifying true sources from real world datasets using a volume-preserving transformation.
Revised: Therefore, two classes are probably sufﬁcient for identifying true sources from real world datasets.

Original: This is consistent with our theory.
Revised: This is consistent with our theory.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We explore a new direction in nonlinear ICA: restrict the mixing function to be volume-preserving, and meanwhile relax the restrictions on the sources.
Revised: We explore a new direction in nonlinear ICA: restrict the mixing function to be volume-preserving, and meanwhile relax the restrictions on the sources.

Original: With some mild conditions, we establish two novel identiﬁability theorems, which guarantee the sources can be identiﬁed up toa point-wise nonlinearity and a point-wise linearity, respectively.
Revised: With some mild conditions, we establish two novel identiﬁability theorems, which guarantee the sources can be identiﬁed up to point-wise nonlinearity and a point-wise linearity, respectively.

Original: The proofs give us some insights for nonlinear ICA: using volume-preserving transformations, the main indeterminacy is simply the rotation of latent variables.
Revised: The proofs give us some insights for nonlinear ICA: if there exists some natural conditions on mixing functions, the main indeterminacy is simply the rotation of latent variables.

Original: These theoretical results provide new paths and new techniques for identiﬁability guarantee.
Revised: This provides new paths and new techniques for identiﬁability guarantee.

Original: To broaden the applicability of our theory, we show in experiments that even most conditions are not satisﬁed, the true sources can be successfully identiﬁed.
Revised: To broaden the applicability of our theory, we show in experiments that even most conditions are not satisﬁed, the true sources can be successfully identiﬁed.

Original: This indicates that there exists stronger identiﬁability theorem with less conditions, which is an appealing future direction to explore.
Revised: This indicates that there exists stronger identiﬁability theorem with less conditions, which is an appealing future direction to explore.

Original: Moreover, we show in experiments that a volume-preserving ﬂow-based model using two classes signiﬁcantly excels the state-of-the-art ICA method (iVAE), and is able to learn interpretable attributes from real world datasets.
Revised: Moreover, we show in experiments that a volume-preserving ﬂow -based model using two classes signiﬁcantly excels the state-of-the-art ICA method (iVAE) and is able to learn interpretable attributes from real world datasets.

Original: These results demonstrate the advantages and applicability of our proposed framework.
Revised: These results demonstrate the advantages and applicability of our proposed framework

Original: A P ROOF OF O UR I DENTIFIABILITY T HEOREM
Revised: A P ROOF OF O UR I DENTIFIABILITY T HEOREM

Original: We ﬁrst prove some lemmas on Euclidean space, and then prove our theorems on Riemannian manifold using these lemmas.
Revised: We ﬁrst prove some lemmas on Euclidean space, and then prove our theorems on Riemannian manifold using these lemmas.

Original: A. L EMMAS ON E UCLIDEAN S PACE
Revised: A. L EMMAS ON E UCLIDEAN S PACE

Original: Let s ∼ p ( s | u ) , s ∈ R n , and there exists a homeomorphism as follows:
Revised: Let s ∼ p ( s | u ) , s ∈ R n , and there exists a homeomorphism as follows:

Original: [Equation]
Revised: [Equation]

Original: Suppose f is volume-preserving:
Revised: Suppose f is volume-preserving:

Original: [Equation]
Revised: [Equation]

Original: This volume-preserving property directly leads to a key corollary: denote the density of z conditioned on u by q ( z | u ) , then p ( s | u ) =
Revised: This volume-preserving property directly leads to a key corollary: denote the density of z conditioned on u by q ( z | u ) , then p ( s | u ) =

Original: q ( f ( s ) | u ) .
Revised: q ( f ( s ) | u ) .

Original: To see this, note that the change of variable rule gives us p ( s | u )
Revised: To see this, note that the change of variable rule gives us p ( s | u )

Original: =
Revised: =

Original: q ( f ( s ) | u ) | det J f ( s ) | , and substituting Eq.
Revised: q ( f ( s ) | u ) | det J f ( s ) | , and substituting Eq.

Original: 12 leads to the conclusion.
Revised: 12 leads to the conclusion.

Original: This is the starting point of our proof.
Revised: This is the starting point of our proof.

Original: Further suppose components of s are independent, and q ( z | u ) is a factorial multivariate Gaussian:
Revised: Further suppose components of s are independent, and q ( z | u ) is a factorial multivariate Gaussian:

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 Consider the model deﬁned by Eq.
Revised: Lemma 1 Consider the model deﬁned by Eq.

Original: 11-Eq.
Revised: 11-Eq.

Original: 14, if f have all second order derivatives, then there exist { θ i, 1 ( u ) , θ i, 2 ( u ) } ni =1 such that
Revised: 14, if f have all second order derivatives, then there exist { θ i, 1 ( u ) , θ i, 2 ( u ) } ni =1 such that

Original: [Equation]
Revised: [Equation]

Original: Proof Note that p ( s | u ) =
Revised: Proof Note that p ( s | u ) =

Original: q ( f ( s ) | u ) , according to Eq.
Revised: q ( f ( s ) | u ) , according to Eq.

Original: 13-Eq.
Revised: 13-Eq.

Original: 14, we have:
Revised: 14, we have:

Original: [Equation]
Revised: [Equation]

Original: Take the logarithm, we have:
Revised: Take the logarithm, we have:

Original: [Equation]
Revised: [Equation]

Original: Because f have all second order derivatives, f i has Taylor expansion:
Revised: Because f have all second order derivatives, f i has Taylor expansion:

Original: f i ( s 0 + ∆ s ) =
Revised: f i ( s 0 + ∆ s ) =

Original: f i ( s 0 ) + ∇ f i ( s 0 ) (cid:62) ∆ s +∆ s (cid:62) H f i ( s 0 )
Revised: f i ( s 0 ) + ∇ f i ( s 0 ) (cid:62) ∆ s +∆ s (cid:62) H f i ( s 0 )

Original: ∆ s + O (∆ s (cid:62) ∆ s ) .
Revised: ∆ s + O (∆ s (cid:62) ∆ s ) .

Original: Let s = s 

Revised: Let s = s

Original: +∆ s in Eq.
Revised: +∆ s in Eq.

Original: 17, and suppose f i ( s 0 )
Revised: 17, and suppose f i ( s 0 )

Original: =without loss of generality, then take the Taylor expansion of f i , we have:
Revised: =without loss of generality, then take the Taylor expansion of f i , we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where η qi = θ qi, 1 ( u ) ∇ f i ( s 0 ) (cid:62) , and C qi is a constant.
Revised: where η qi = θ qi, 1 ( u ) ∇ f i ( s 0 ) (cid:62) , and C qi is a constant.

Original: Note that the left hand side of the equation above does not contain cross terms { ∆ s j ∆ s k } j (cid:54)
Revised: Note that the left hand side of the equation above does not contain cross terms { ∆ s j ∆ s k } j (cid:54)

Original: =
Revised: =

Original: k , while the right hand side does.
Revised: k , while the right hand side does.

Original: Therefore, the coefﬁcients of cross terms on the right hand side should be zeros:
Revised: Therefore, the coefﬁcients of cross terms on the right hand side should be zeros:

Original: [Equation]
Revised: [Equation]

Original: Let θ i, 1 ( u ) ≡ θ q i, 1 ( u ) and θ i, 2 ( u ) ≡ 2 θ q i, 2 ( u ) , then the conclusion is obtained.
Revised: Let θ i, 1 ( u ) ≡ θ q i, 1 ( u ) and θ i, 2 ( u ) ≡ 2 θ q i, 2 ( u ) , then the conclusion is obtained.

Original: [Equation]
Revised: [Equation]

Original: Remark:
Revised: Remark:

Original: Denote the mean of q i by µ i , and the variance by σ 2 i , then θ qi, 1 = − µ i σ 

Revised: Denote the mean of q i by µ i , and the variance by σ 2 i , then θ qi, 1 = − µ i σ

Original: i and θ qi, 2 = 12 σ 2 i .
Revised: i and θ qi, 2 = 12 σ 2 i .

Original: Without loss of generality, we can set µ i = 0 , ∀ i , and hence θ i, 1 = 0 , ∀ i .
Revised: Without loss of generality, we can set µ i = 0 , ∀ i , and hence θ i, 1 = 0 , ∀ i .

Original: Therefore, Eq. 15 can be further simpliﬁed as
Revised: Therefore, Eq. 15 can be further simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 Suppose all conditions in Lemma 1 are satisﬁed, and there exists two distinct values of u , denoted by u (1) and u (2) , such that the following conditions are also satisﬁed:
Revised: Lemma 2 Suppose all conditions in Lemma 1 are satisﬁed, and there exists two distinct values of u , denoted by u (1) and u (2) , such that the following conditions are also satisﬁed:

Original: i) µ i ( u (1) )
Revised: i) µ i ( u (1) )

Original: =
Revised: =

Original: µ i ( u (2) ) , ∀ i ∈ [ n ] ; ii) (cid:110) σ 1 ( u (2) )
Revised: µ i ( u (2) ) , ∀ i ∈ [ n ] ; ii) (cid:110) σ 1 ( u (2) )

Original: σ 1 ( u (1) ) , · · · , σ n ( u (2) )
Revised: σ 1 ( u (1) ) , · · · , σ n ( u (2) )

Original: σ n ( u (1) ) (cid:111) are distinct.
Revised: σ n ( u (1) ) (cid:111) are distinct.

Original: Then ∀ s ∈ R n , the Jacobian J f ( s ) = (cid:16) ∂f i ∂ s j ( s ) (cid:17) n × n is a generalized permutation matrix.
Revised: Then ∀ s ∈ R n , the Jacobian J f ( s ) = (cid:16) ∂f i ∂ s j ( s ) (cid:17) n × n is a generalized permutation matrix.

Original: Proof Without loss of generality, let µ i ( u (1) )
Revised: Proof Without loss of generality, let µ i ( u (1) )

Original: = 0 , ∀ i ∈ [ n ] , according to Lemma 1 and condition i), we have
Revised: = 0 , ∀ i ∈ [ n ] , according to Lemma 1 and condition i), we have

Original: [Equation]
Revised: [Equation]

Original: Let Σ ( u , s ) ≡ diag (cid:18)(cid:80) ni
Revised: Let Σ ( u , s ) ≡ diag (cid:18)(cid:80) ni

Original: =

Revised: =

Original: θ i, 2 ( u ) (cid:16) ∂f i ∂ s 1 ( s ) (cid:17) 2 , · · · , (cid:80) ni
Revised: θ i, 2 ( u ) (cid:16) ∂f i ∂ s 1 ( s ) (cid:17) 2 , · · · , (cid:80) ni

Original: =

Revised: =

Original: θ i, 2 ( u ) (cid:16) ∂f i ∂ s n ( s ) (cid:17) 2 (cid:19) and Λ ( u ) ≡ diag ( θ 1 , 2 ( u ) , · · · , θ n, 2 ( u )) , then the equation above can be written as
Revised: θ i, 2 ( u ) (cid:16) ∂f i ∂ s n ( s ) (cid:17) 2 (cid:19) and Λ ( u ) ≡ diag ( θ 1 , 2 ( u ) , · · · , θ n, 2 ( u )) , then the equation above can be written as

Original: [Equation]
Revised: [Equation]

Original: Due to θ i, 2 ( u ) = 1 σ 2 i ( u ) > 0 , Σ ( u , s ) and Λ ( u ) are always positive-deﬁned, hence the equations above have square roots on both the two hand sides as follows:
Revised: Due to θ i, 2 ( u ) = 1 σ 2 i ( u ) > 0 , Σ ( u , s ) and Λ ( u ) are always positive deﬁnite, hence the equations above have square roots on both the two hand sides as follows:

Original: [Equation]
Revised: [Equation]

Original: where U ( u ) is orthogonal matrices, i.e. U ( u ) (cid:62) U ( u )
Revised: where U ( u ) is orthogonal matrices, i.e. U ( u ) (cid:62) U ( u )

Original: =
Revised: =

Original: I .
Revised: I .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This leads to
Revised: This leads to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Σ
Revised: Σ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Both the two hand sides of the equation above can be viewed as the singular value decomposition (SVD) of Σ ( u (2) , s ) 1 / 2 Σ ( u (1) , s ) − 1 / 2 .
Revised: Both the two hand sides of the equation above can be viewed as the singular value decomposition (SVD) of Σ ( u (2) , s ) 1 / 2 Σ ( u (1) , s ) − 1 / 2 .

Original: Note that due to condition ii), the diagonal entries of Λ ( u (2) ) 1 / 2 Λ ( u (1) ) − 1 / 2 are distinct.
Revised: Note that due to condition ii), the diagonal entries of Λ ( u (2) ) 1 / 2 Λ ( u (1) ) − 1 / 2 are distinct.

Original: Based on this, according to the uniqueness of SVD, U ( u (1) ) is the composition of a permutation matrix and a signature matrix.
Revised: Based on this, according to the uniqueness of SVD (Kong et al., 2017), U ( u (1) ) is the composition of a permutation matrix and a signature matrix.

Original: Substitute such decomposition of U ( u (1) ) into Eq.
Revised: Substitute such decomposition of U ( u (1) ) into Eq.

Original: 24, we ﬁnally conclude that J f ( s ) is a generalized permutation matrix.
Revised: 24, we ﬁnally conclude that J f ( s ) is a generalized permutation matrix.

Original: (cid:3)
Revised: [Equation]

Original: Remarks: Since J f ( s ) is a generalized permutation matrix, in each row and each column it has exactly one nonzero entry.
Revised: Remarks: Since J f ( s ) is a generalized permutation matrix, in each row and each column it has exactly one nonzero entry.

Original: Without loss of generality, we can assume ∂f i ∂
Revised: Without loss of generality, we can assume ∂f i ∂

Original: s i ( s ) (cid:54)
Revised: s i ( s ) (cid:54)

Original: = 0 , while ∂f i ∂ s j ( s )
Revised: = 0 , while ∂f i ∂ s j ( s )

Original: = 0 , ∀ j (cid:54)
Revised: = 0 , ∀ j (cid:54)

Original: =
Revised: =

Original: i .
Revised: i .

Original: Therefore, f i ( s ) is exactly the function of s i and is independent of { s j } j (cid:54)
Revised: Therefore, f i ( s ) is exactly the function of s i and is independent of { s j } j (cid:54)

Original: =
Revised: =

Original: i , and hence it can be denoted by f i ( s i ) .
Revised: i , and hence it can be denoted by f i ( s i ) .

Original: Therefore, under some mild conditions in Lemma 2, z is identical to s up to a nonlinear point-wise transformation (and a permutation).
Revised: Therefore, under some mild conditions in Lemma 2, z is identical to s up to a nonlinear point-wise transformation (and a permutation).

Original: A. T HEOREMS ON R IEMANNIAN M ANIFOLD
Revised: A. T HEOREMS ON R IEMANNIAN M ANIFOLD

Original: Suppose data points are distributed on a Riemannian manifold M which is homeomorphic to R n .
Revised: Suppose data points are distributed on a Riemannian manifold M which is homeomorphic to R n .

Original: The homeomorphic can be denoted by
Revised: The homeomorphic can be denoted by

Original: [Equation]
Revised: [Equation]

Original: which is further assumed to be volume-preserving:
Revised: which is further assumed to be volume-preserving:

Original: [Equation]
Revised: [Equation]

Original: The density of s conditioned on u is denoted by p ( s | u ) , which is assumed to be factorial, i.e. { s i } ni = are independent:
Revised: The density of s conditioned on u is denoted by p ( s | u ) , which is assumed to be factorial, i.e. { s i } ni = are independent:

Original: [Equation]
Revised: [Equation]

Original: Suppose the data variable x can be encoded into a vector of latent variables z with another homeomorphism, denoted by
Revised: Suppose the data variable x can be encoded into a vector of latent variables z with another homeomorphism, denoted by

Original: [Equation]
Revised: [Equation]

Original: where the homeomorphism is also volume-preserving:
Revised: where the homeomorphism is also volume-preserving:

Original: [Equation]
Revised: [Equation]

Original: and conditioned on u , z follows a factorial multivariate Gaussian, denoted by
Revised: and conditioned on u , z follows a factorial multivariate Gaussian, denoted by

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 ( Nonlinear Identiﬁability ) Assume data points are sampled from a generative model deﬁned according to Eq.
Revised: Theorem 1 ( Nonlinear Identiﬁability ) Assume data points are sampled from a generative model deﬁned according to Eq.

Original: 26-Eq. 31, and there exist two distinct observations of u , denoted by u (1) and u (2) , such that the following holds:
Revised: 26-Eq. 31, and there exist two distinct observations of u , denoted by u (1) and u (2) , such that the following holds:

Original: i)
Revised: i)

Original: Both f and g have all second order derivatives.
Revised: Both f and g have all second order derivatives.

Original: ii) µ i ( u (1) )
Revised: ii) µ i ( u (1) )

Original: = µ i ( u (2) ) , ∀ i ∈ [ n ] . iii) (cid:110) σ 1 ( u (2) )
Revised: = µ i ( u (2) ) , ∀ i ∈ [ n ] . iii) (cid:110) σ 1 ( u (2) )

Original: σ 1 ( u (1) ) , · · · , σ n ( u (2) )
Revised: σ 1 ( u (1) ) , · · · , σ n ( u (2) )

Original: σ n ( u
Revised: σ n ( u

Original: (1) ) (cid:111) are distinct.
Revised: (1) ) (cid:111) are distinct.

Original: Then g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.
Revised: Then g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.

Original: Proof Note that g ◦ f : s (cid:55)→ z is a homeomorphism which is volume-preserving | det J g ◦ f ( s ) | =and has all second order derivatives.
Revised: Proof Note that g ◦ f : s (cid:55)→ z is a homeomorphism which is volume-preserving | det J g ◦ f ( s ) | =and has all second order derivatives.

Original: According to Lemma 2, we can conclude that g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.
Revised: According to Lemma 2, we can conclude that g ◦ f is a composition of a point-wise nonlinear transformation and a permutation.

Original: (cid:3)
Revised: (cid:3)

Original: Theorem 2 ( Linear Identiﬁability ) Assume the hypotheses of Theorem 1 hold, and p ( s | u (1) ) and p ( s | u (2) ) are multivariate Gaussians.
Revised: Theorem 2 ( Linear Identiﬁability ) Assume the hypotheses of Theorem 1 hold, and p ( s | u (1) ) and p ( s | u (2) ) are multivariate Gaussians.

Original: Then g ◦ f is a composite of a point-wise linear transformation and a permutation.
Revised: Then g ◦ f is a composite of a point-wise linear transformation and a permutation.

Original: Proof Let h =
Revised: Proof Let h =

Original: g ◦ f .
Revised: g ◦ f .

Original: According to Theorem 1, h is a composite of point-wise nonlinear transformation and a permutation, and hence without loss of generality, we write
Revised: According to Theorem 1, h is a composite of point-wise nonlinear transformation and a permutation, and hence without loss of generality, we write

Original: [Equation]
Revised: [Equation]

Original: Since p ( s | u ) is a multivariate Gaussian, let p ( s | u ) =
Revised: Since p ( s | u ) is a multivariate Gaussian, let p ( s | u ) =

Original: (cid:81) ni
Revised: (cid:81) ni

Original: =1 1 Z pi exp (cid:0) − θ pi, 1 ( u ) s i − θ pi, 2 ( u ) s 2 i (cid:1) .
Revised: =1 1 Z pi exp (cid:0) − θ pi, 1 ( u ) s i − θ pi, 2 ( u ) s 2 i (cid:1) .

Original: Substitute this expression and the equation above into Eq.
Revised: Substitute this expression and the equation above into Eq.

Original: 17, we have
Revised: 17, we have

Original: [Equation]
Revised: [Equation]

Original: (33) where C i is a constant.
Revised: (33) where C i is a constant.

Original: This obviously leads to
Revised: This obviously leads to

Original: [Equation]
Revised: [Equation]

Original: To conclude, g ◦ f is a composition of a point-wise linear transformation and a permutation.
Revised: To conclude, g ◦ f is a composition of a point-wise linear transformation and a permutation.

Original: (cid:3)
Revised: (cid:3)

Original: B A C LASS OF I DENTIFIABLE NVP M IXING F UNCTIONS
Revised: B A C LASS OF I DENTIFIABLE NVP M IXING F UNCTIONS

Original: Even if the mixing function is non-volume-preserving (NVP), it still leads to identiﬁability when its
Revised: Even if the mixing function is non-volume-preserving (NVP), it still leads to identiﬁability when its

Original: Jacobian has the following form:
Revised: Jacobian has the following form:

Original: [Equation]
Revised: [Equation]

Original: where h i is arbitrary function satisfying |
Revised: where h i is arbitrary function satisfying |

Original: h
Revised: h

Original: i ( z i ) | > 0 .
Revised: i ( s i ) | > 0 .

Original: Obviously this class of function is a extension of volume-preserving transformations.
Revised: Obviously this class of function is a extension of volume-preserving transformations.

Original: To see this, note that when | h i ( z i ) | = 1 , this class of function degenerate into volume-preserving transformations.
Revised: To see this, note that when | h i ( s i ) | = 1 , this class of function degenerate into volume-preserving transformations.

Original: Substitute the equation above into the change of variable rule p ( s | u ) =
Revised: Substitute the equation above into the change of variable rule p ( s | u ) =

Original: q ( f ( s ) | u ) | det J f ( s ) | , and take the logarithm, we have
Revised: q ( f ( s ) | u ) | det J f ( s ) | , and take the logarithm, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using this equation, the proof of Lemma 1 still holds, because the in the proof we only use the fact that the left hand side of Eq.
Revised: Using this equation, the proof of Lemma 1 still holds, because the in the proof we only use the fact that the left hand side of Eq.

Original: 17 has no cross terms, which is also the property of the equation above.
Revised: 17 has no cross terms, which is also the property of the equation above.

Original: Therefore, the class of mixing functions deﬁned above also leads to identiﬁability, even if it is nonvolume-preserving.
Revised: Therefore, the class of mixing functions deﬁned above also leads to identiﬁability, even if it is nonvolume-preserving.

Original: This means that there exist some non-trival extensions of our theory.
Revised: This means that there exist some non-trival extensions of our theory.

Original: C S IMPLIFIED L OSS F UNCTION
Revised: C S IMPLIFIED L OSS F UNCTION

Original: Consider when µ i ( u ) and σ i ( u ) are trainable, their optimal solutions  have the closed forms as follows:[Equation]
Revised: Consider when µ i ( u ) and σ i ( u ) are trainable, their optimal solutions of Eq. 9 have the closed forms as follows:

Original: [Equation]
Revised: [Equation]

Original: where k
Revised: where k

Original: = 1 , 2 .
Revised: = 1 , 2 .

Original: Note that the optimal solutions depend on the parameters θ , hence when given θ , we denote µ ∗ i ( u ( k ) ) and σ ∗ i ( u ( k ) ) by µ i ( u ( k ) , θ ) and σ i ( u ( k ) ; θ ) respectively.
Revised: Note that the optimal solutions depend on the parameters θ , hence when given θ , we denote µ ∗ i ( u ( k ) ) and σ ∗ i ( u ( k ) ) by µ i ( u ( k ) , θ ) and σ i ( u ( k ) ; θ ) respectively.

Original: Substitute these optimal solutions to the loss function above, we ﬁnally obtain:
Revised: Substitute these optimal solutions to the loss function above, we ﬁnally obtain:

Original: [Equation]
Revised: [Equation]

Original:  In experiments, we only sample a mini-batch B from the full dataset D for each iteration, hence in this case σ i ( u ( k ) ; θ ) = (cid:113) E ( x , u ( k ) ) ∈B (cid:2)
Revised: [Equation] In experiments, we only sample a mini-batch B from the full dataset D for each iteration, hence in this case σ i ( u ( k ) ; θ ) = (cid:113) E ( x , u ( k ) ) ∈B (cid:2)

Original: g i ( x ; θ ) − µ i ( u ( k ) ; θ ) (cid:3) 2 , which is the biased standard deviation of { g i ( x ; θ ) } ( x , u ( k ) ) ∈B .
Revised: g i ( x ; θ ) − µ i ( u ( k ) ; θ ) (cid:3) 2 , which is the biased standard deviation of { g i ( x ; θ ) } ( x , u ( k ) ) ∈B .

Original:  D D ETAILS A BOUT THE S YNTHESIZED I MAGE D ATASET
Revised: C. D IMENSION R EDUCTION

Revised: Here we will demonstrate that the simpliﬁed loss above leads to dimension reduction. This is a vital property for identiﬁability. On the one side, dimension reduction enable the learning algorithm to estimate the unknown dimensionality of data manifold.

Revised: On the other, if we use a ﬂow-based model to identify the true sources, then dimension reduction of latent variables z is necessary, as in ﬂow-based models the dimensionality of z ( d ) is far more than the number of true sources ( n ).

Revised: In a ﬂow-based model, the simpliﬁed loss is

Revised: [Equation]

Revised: Hence minimizing the simpliﬁed loss is equivalent to minimizing (cid:81) di =1 σ i ( u ( k ) ; θ ) .

Revised: To minimizing this term, most latent variables should have almost zero standard deviations.

Revised: Note that in implementation, the standard deviations of latent variables could not be exact zero, and not all standard deviations can be almost zero.

Revised: First, in ﬂow-based models, the data points are usually augmented by adding small noise, and hence the dimensionality of augmented data manifold is exactly d .

Revised: As a result, the dimensionality of the corresponding latent space is also d , hence exact zero standard deviation is not possible due to the reversibility of ﬂow-based models.

Revised: Moreover, as we assume the transformation g is volume-preserving, the entropy of data distribution and latent distribution should be equal, and hence there must exist some latent variables with high standard deviations.

Revised: Therefore, the simpliﬁed loss will assign almost zero standard deviations to most latent variables while preserving some high standard deviations.

Revised: A natural solution is to assign high standard deviations to meaningful latent variables, and assign almost zero standard deviations to redundant dimensions.

Revised: This is exactly the dimension reduction ability of the simpliﬁed loss.

Revised: C. I NDEPENDENCE E NHANCEMENT

Revised: Next we will prove that the simpliﬁed loss can enhance the independence of different components of z .

Revised: This is a desirable property, as p ( z | u ) should be factorial according to our theory.

Revised: Our proof is based on Hadamard’s inequality (Cover & Gamal, 1983): for any n × n matrix K , det K ≤ (cid:81)

Revised: i K i,i with equality

Revised: iff K i,j = 0 , ∀ i (cid:54) =

Revised: j .

Revised: Let K

Revised: =

Revised: Cov ( z , z ) , then we have det Cov

Revised: [Equation]

Revised: z , z ) ≤ (cid:81)

Revised: i

Revised: Var ( z i ) .

Revised: Then the simpliﬁed loss has a lower bound as follows:

Revised: [Equation]

Revised: where the subscripts u ( k ) represent the conditions.

Revised: When the simpliﬁed loss is optimized, the gap between it and its lower bound will become zero, and hence the equality holds.

Revised: This means Cov ( z i , z j ) = 0 , ∀ i (cid:54) =

Revised: j .

Revised: Therefore, minimizing the simpliﬁed loss can enhance the independence of different latent variables by reducing correlations.

Revised: In addition, note that our derivation above does not depend on the Gaussianity of q ( z | u ) , and hence our conclusion is general.

Revised: D D ETAILS A BOUT THE A RTIFICIAL AND S YNTHESIZED D ATASETS

Original: [Table]
Revised: [Table]

Original: The synthetic image dataset “Triangles” is a set of gray-scale 32 × 32 images, in which eachimage contains a shape of right triangle.
Revised: To generate the the artiﬁcial data, following (Sorrenson et al., 2020), we synthesize the sources and observed data by two steps: i) First, two sources are sampled from a 2-dim mixture of two Gaussians with their mean both (0 . 0 , 0 . 0) , and with variances (1 . 0 , 0 . 5) and (0 . 5 , 1 . 0) , respectively. From each

Original: As shown in Fig.
Revised: Gaussian, 5 , 000 points are sampled. Then an 8-dim standard Gaussian noise scaled by 0 . 01 is concatenated with them.

Original: 6 the factors of variation includes rotation, width (length of the horizontal edge), height (length of the vertical edge) and gray level.
Revised: ii) The observed data is generated from the 10-dim sources, by a randomly initialized GIN (Sorrenson et al., 2020) with 8 fully connected coupling blocks.

Original: The value ranges of four factors are set as [ − π, π ] , [1 , 32] , [1 , 32] and [0 , 255] , respectively. We view these factors as the sources in our theory.
Revised: To quantitatively test our method on image datasets, we have synthesized a gray-scale image dataset called “Triangles”. Speciﬁcally, we choose to generate a grey-scale 32 × 32 image dataset of 2-D shapes similar with dSprites (Matthey et al., 2017) as shown in Fig. 6, in which all shapes are right triangles generated from

Original: According to our requirement, in each class these factors follow a factorial multivariate Gaussian.
Revised: According to our requirement, in each class these factors follow a factorial multivariate Gaussian.

Original: To satisﬁed condition (iii) in Theorem 1, we set the variance of each factor as a proper random value, and so do the mean because we found in experiments that the condition (ii) in Theorem 1 is not necessary.
Revised: To satisﬁed condition (iii) in Theorem 1, we set the variance of each factor as a proper random value, and so do the mean because we found in experiments that the condition (ii) in Theorem 1 is not necessary.

Original: The particular parameters of the four factors are shown in Table
Revised: The particular parameters of the four factors are shown in Table

Original: The generating process of each image consists of three steps: i) for each class, randomly select four means and four variances according to Table 3, and obtain a 4-dim Gaussian; ii) sample one point from the 4-dim Gaussian, which corresponds to a right triangle; iii) judge what pixels locate inside the triangle according to the given rotation, width and height, and then assign them with the given gray level.
Revised: The generating process of each image consists of three steps: i) for each class, randomly select four means and four variances according to Table 3, and obtain a 4-dim Gaussian; ii) sample one point from the 4-dim Gaussian, which corresponds to a right triangle; iii) judge what pixels locate inside the triangle according to the given rotation, width and height, and then assign them with the given gray level.

Original: E Q UALITATIVE R ESULTS OF A BLATION S TUDY
Revised: E Q UALITATIVE R ESULTS OF A BLATION S TUDY

Original: Here we show the qualitative results of our ablation study (see Fig. 7) to support our discussion in an intuitive way.
Revised: Here we show the qualitative results of our ablation study (see Fig. 7) to support our discussion in an intuitive way.

Original: In these ﬁgures, reconstruction is the estimated latent variables with top two standard deviations, as Sorrenson et al.
Revised: In these ﬁgures, reconstruction is the estimated latent variables with top two standard deviations, as Sorrenson et al.

Original: (2020) reported that estimated latent variables with higher standard deviations are probably more meaningful.
Revised: (2020) reported that estimated latent variables with higher standard deviations are probably more meaningful.

Original: Spectrum is the sorted standard deviations of latent variables.
Revised: Spectrum is the sorted standard deviations of latent variables.

Original: The spectrum of the estimated latent variables is in black, while the spectrum of true latent variables is in grey.
Revised: The spectrum of the estimated latent variables is in black, while the spectrum of true latent variables is in grey.

Revised: F Q UALITATIVE R ESULTS ON C ELEB A

Revised: To further show the performance of our framework on real-world datasets, we conduct experiments on CelebA (Liu et al., 2015) using GIN.

Revised: Since our framework requires two distinct classes, we use all images in CelebA as one class, and their mirror images as another class.

Revised: Such setting ensures the two obtained classes are distinct and have non-zero overlap.

Original: [Figure]
Revised: Similar with experiments on MNIST, we pick latent dimensions with high standard deviations as estimated latent variables, and plot them to show their corresponding underlying factors. T As shown in Fig. 8, the estimated latent variables with top 4 standard deviations are highly interpretable, corresponding to lighting on the left, lighting on the right, skin color, background brightness, hair (bang) and grin, respectively.

Revised: These results further demonstrate that our framework is probably able to identify the true sources from real-world datasets and hence support our theory

Revised: [Figure]