Original: O N THE I DENTIFIABILITY OF N ONLINEAR ICA WITH U NCONDITIONAL P RIORS
Revised: O N THE I DENTIFIABILITY OF N ONLINEAR ICA WITH U NCONDITIONAL P RIORS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Nonlinear independent component analysis (ICA) aims to recover the underlying marginally independent latent sources from their observable nonlinear mixtures.
Revised: Nonlinear independent component analysis (ICA) aims to recover the underlying marginally independent latent sources from their observable nonlinear mixtures

Original: The identifiability of nonlinear ICA is a major unsolved problem in unsupervised learning.
Revised: The identifiability of nonlinear ICA is a major unsolved problem in unsupervised learning.

Original: Recent breakthroughs reformulate the standard marginal independence assumption of sources as conditional independence given some auxiliary variables (e.g., class labels) as weak supervision or inductive bias.
Revised: Recent breakthroughs reformulate the standard marginal independence assumption of sources as conditional independence given some auxiliary variables (e g., class labels) as weak supervision or inductive bias.

Original: However, the modified setting is not  applicable in many scenarios that do not have auxiliary variables.
Revised: However, the modified setting might not be applicable in many scenarios that do not have auxiliary variables.

Original: We explore an alternative path and considerinstead only assumptions on the mixing process, such as the pairwise orthogonality among the columns of the
Revised: We explore an alternative path and consider only assumptions on the mixing process, such as independent influences .

Original: Jacobian of the mixing function.

Original: We show  that  marginally independent latent sources can be identified from strongly nonlinear mixtures up to a component-wise  transformation and a permutation, thus providing, to the best of our knowledge, a first full identifiability result of nonlinear ICA without auxiliary variables.
Revised: We show under these assumptions that the marginally independent latent sources can be identified from the nonlinear mixtures up to a component-wise (linear) transformation and a permutation, thus providing an identifiability result of nonlinear ICA without auxiliary variables.

Original: We provide an estimation method and validate the theoretical results experimentally.
Revised: We provide an estimation method and validate the theoretical results experimentally

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nonlinear independent component analysis (ICA) is a fundamental problem in unsupervised representation learning.
Revised: Nonlinear independent component analysis (ICA) is a fundamental problem in unsupervised representation learning.

Original: It generalizes linear ICA
Revised: It generalizes linear ICA

Original: (Hyv ¨ arinen et al., 2001; Comon, 1994) to identify latent sources from observations, which are assumed to be an invertible nonlinear mixture of them.
Revised: (Hyv ¨ arinen et al., 2001; Comon, 1994) to identify latent sources from observations, which are assumed to be an invertible nonlinear mixture of the sources.

Original: For an observed vector x , nonlinear ICA expresses it as x = f ( s ) , where f is an unknown invertible mixing function, and s is a latent random vector representing the marginally independent sources.
Revised: For an observed vector x , nonlinear ICA expresses it as x = f ( s ) , where f is an unknown invertible mixing function, and s is a latent random vector representing the marginally independent sources.

Original: The goal is to recover function f as well as sources s from the observed mixture x .
Revised: The goal is to recover function f as well as sources s from the observed mixture x up to certain indeterminacies.

Original: Whilethere is an increasing interest in nonlinear ICA  in various tasks, such as disentanglement (Lachapelle et al., 2021), its identifiability remains a major obstacle for providing a principled framework.
Revised: While nonlinear ICA is of general interest in a variety of tasks for decades, such as disentanglement (Lachapelle et al., 2022) and unsupervised learning (Oja, 2002), its identifiability remains a major obstacle for providing a principled framework.

Original: The key difficulty is that, without additional assumptions, there exist infinite ways to transform the observations into independent components while still mixed w.r.t.
Revised: The key difficulty is that, without additional assumptions, there exist infinite ways to transform the observations into independent components while still mixed w.r.t.

Original: the sources (Hyv ¨ arinen & Pajunen, 1999).
Revised: the sources (Hyv ¨ arinen & Pajunen, 1999).

Original: To deal with this challenge, existing works introduce the auxiliary variable u (e.g.,  label, domain index, timestamp) and assume that sources are conditionally independent given u (Hyvarinen &
Revised: To deal with this challenge, existing works introduce the auxiliary variable u (e.g., class label, domain index, time dependency) and assume that sources are conditionally independent given u (Hyv¨arinen &

Original: Morioka, 2017; Hyvarinen et al., 2019; Khemakhem et al., 2020).
Revised: Morioka, 2017; Hyv¨arinen et al., 2019; Khemakhem et al., 2020).

Original: Most of them require the auxiliary variable to be observable, while Hyvarinen & Morioka (2016) and H¨alv¨a & Hyvarinen (2020) are exceptions.
Revised: Most of them require the auxiliary variable to be observable, while clustering-based methods (Hyv¨arinen & Morioka, 2016) and those for time series (H¨alv¨a & Hyv¨arinen, 2020; H¨alv¨a et al., 2021) are exceptions.

Original: These works impose mild restrictions on the mixing process but require many distinct values of u (e.g., number of classes) forfull identifiability, which might restrict their practical applicability.
Revised: These works impose mild restrictions on the mixing process but require many distinct values of u (e.g., number of classes) for identifiability, which might restrict their practical applicability.

Original: Motivated by this, Yang et al.
Revised: Motivated by this, Yang et al.

Original: (2022) reduce the number of required distinct values by strengthening  functional assumptions on the mixing process.
Revised: (2022) reduce the number of required distinct values by strengthening the functional assumptions on the mixing process.

Original: As described, all these results are based on conditional independence, instead of the standard marginal independence assumption, of the sourcesin order to achieve identifiability in specific tasks, i.e., those with auxiliary variables as weak supervision.
Revised: As described, all these results are based on conditional independence, instead of the standard marginal independence assumption, of the sources to achieve identifiability in specific tasks, i.e., those with auxiliary variables as weak supervision.

Original: The identifiability of the original setting focusing on marginal independence, which is crucial for unsupervised learning, stays unsolved.
Revised: The identifiability of the original setting focusing on marginal independence, which is crucial for unsupervised learning, stays unsolved.

Original: Instead of introducing auxiliary variables, there exists the possibility to rely on further restrictions on the mixing functions to achieve identifiability, but limited results are available in the literature over the past two decades and are often based on overly restrictive conditions.
Revised: Instead of introducing auxiliary variables, there exists the possibility to rely on further restrictions on the mixing functions to achieve identifiability, but limited results are available in the literature over the past two decades.

Original: With the assumption of conformal transformation, Hyv¨arinen & Pajunen (1999) proved identifiability of the nonlinear ICA solution up to rotational indeterminacy for only two  sources.
Revised: With the assumption of conformal transformation, Hyv¨arinen & Pajunen (1999) proved identifiability of the nonlinear ICA solution up to rotational indeterminacy for only two and sources.

Original: Another result is shown on the mixture that consists of component-wise nonlinearities added to a linear mixture (Taleb & Jutten, 1999),
Revised: Another result is shown for the mixture that consists of component-wise nonlinearities added to a linear mixture (Taleb & Jutten, 1999).

Original: Under at the and which relies on the linearity assumption.

Original: MND (Zhang & Chan, 2008) improves the performance of separation by preferring solutions with high linearity, while no identifiability result is given.
Revised: MND (Zhang & Chan, 2008) improves the performance of separation by preferring solutions with high linearity, while no identifiability result is given.

Original: In this work, we aim to show full identifiability of nonlinear ICA with unconditional priors under specific conditions.
Revised: In this work, we aim to show the identifiability of nonlinear ICA with unconditional priors under specific conditions.

Original: We show that without any kind of auxiliary variables, the latent sources can be identified from nonlinear mixtures up to a component-wise invertible transformation and a permutation under assumptions on the mixing process.
Revised: We show that without any kind of auxiliary variables, the latent sources can be identified from nonlinear mixtures up to a component-wise invertible transformation and a permutation under assumptions on the mixing process.

Original: Specifically, we study the following conditions on the mixing process: (1) pairwise orthogonal columns of  the Jacobian matrix, and (2) factorial Jacobian determinant (e.g., volume-preserving transformation).
Revised: Specifically, we study the following conditions on the mixing process: (1) the influence of each source on the observed variables is independent of each other in a non-statistical sense, and (2) factorial Jacobian determinant (e.g., volume-preserving transformation).

Original: Different from existing works relying on weak supervision of auxiliary variables (observable or not), we establish a new identifiability result in a fully unsupervised setting, which is within the original setting of ICA that is based on marginal independence of the latent sources.
Revised: Different from existing works relying on weak supervision of auxiliary variables (observable or not), we establish an identifiability result in a fully unsupervised setting, which is within the original setting of ICA that is based on marginal independence of the latent sources.

Original: In particular, this is consistent with previous results on ruling out some spurious solutions (Gresele et al., 2021) and reducing the number of required labels (Yang et al., 2022) by restricting mixing function f .
Revised: In particular, our work is consistent with previous results on ruling out specific spurious solutions (Gresele et al., 2021) and reducing the number of required values of auxiliary variable (Yang et al., 2022) by restricting mixing function f .

Original: Besides, we show that the component-wise indeterminacy can be further reduced to the component-wise linear transformations if the true sources are assumed to be Gaussian.
Revised: Besides, we show that the component-wise indeterminacy can be further reduced to the component-wise linear transformations if the true sources are assumed to be Gaussian.

Original: This work establishes a mathematical principle underlying unsupervised representation learning, opening up the possibility of  identifiable nonlinear ICA under the standard marginal independence assumption, while not restricting it to the two-dimensional or the post-nonlinear case.
Revised: This work opens up the possibility of fully identifiable nonlinear ICA under the standard marginal independence assumption. We validate our theoretical claims experimentally.  N ONLINEAR ICA WITH UNCONDITIONAL PRIORS In this section, we present two identifiability results for nonlinear ICA with unconditional priors.

Original: We further develop an estimation method and validate our theoretical claims experimentally.
Revised: Instead of introducing any type of auxiliary variables or side information, we consider conditions only on the mixing process, which are rather natural in specific real-world scenarios.

Original: N ONLINEAR ICA WITH UNCONDITIONAL PRIORS 2. I DENTIFIABILITY OF NONLINEAR ICA
Revised: 2. I DENTIFIABILITY OF NONLINEAR ICA

Original: We consider the following data-generating process of nonlinear ICA
Revised: We consider the following data-generating process of nonlinear ICA

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: where n denotes the number of latent sources, x = ( x 1 , . . . , x n ) denotes the observed random vector, s =

Original: [Equation]
Revised: ( s 1 , . . .

Original: where x denotes the observed random vector, s  is the latent random vector representing the marginally independent sources, p s i is the marginal probability density function (pdf) of s i ,
Revised: , s n ) is the latent random vector representing the marginally independent sources, p s i is the marginal probability density function (pdf) of the i -th source

Original:  p s is the joint pdf, and f : s → x denotes a nonlinear mixing function.
Revised: s i , p s is the joint pdf, and f : s → x denotes a nonlinear mixing function.

Original: Note that different from previous works for identifiable nonlinear ICA, sources s do not need to be mutually conditionally independent given an auxiliary variable.
Revised: Different from previous works for identifiable nonlinear ICA, sources s do not need to be mutually conditionally independent given an auxiliary variable.

Original: Instead, our setting is consistent with the traditional ICA problem and thus unifies ICA with auxiliary variables as a special case.
Revised: Instead, our setting is consistent with the traditional ICA problem based on a general marginal independence assumption of sources.

Original: Besides, no particular form of the source distribution is assumed at this point, which is similar to (Yang et al., 2022) and weaker than works assuming an exponential family (Hyvarinen et al., 2019;
Revised: Besides, no particular form of the source distribution is assumed at this point, which is similar to (Yang et al., 2022) and different from the works that directly assume an exponential family (Hyv ¨ arinen et al., 2019;

Original: H ¨ alv ¨ a & Hyvarinen, 2020; Khemakhem et al., 2020; Lachapelle et al., 2021).
Revised: H ¨ alv ¨ a & Hyv ¨ arinen, 2020; Lachapelle et al., 2022).

Original: Following previous works, we assume the  function f is invertible and twice differentiable. We further assume that the column vectors of the Jacobian matrix of f are pairwise orthogonal, i.e.,
Revised: Let g : x → ˆ s be the estimated unmixing function with ˆ s =

Original: [Equation]
Revised: (ˆ s 1 , . . .

Original: where O ( s ) is an orthogonal matrix and D ( s  ) is a diagonal matrix.
Revised: , ˆ s n ) being the marginally independent estimated sources.

Original: Regarding the estimating model, similarly, we set thecolumns of Jacobian  ofestimating function g − 1 : x → ˆ s orthogonal to each other and with equivalent euclidean norm, i.e., J g− 1 ( x ) =
Revised: Denoting by J g the Jacobian matrix of g , we set its columns to be orthogonal to each other and have equivalent euclidean norm, which can be written as J g ( x ) =

Original: O ( x ) λ ( x ) , where  λ ( x ) is a scalar.
Revised: O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar.

Original: Note that restricting D to λ in the estimating process does not affect the component-wise similarity between  the recovered and true sources.
Revised: This is consistent with the pairwise uncorrelatedness between columns of the Jacobian after centering.

Original: We assume that the distribution of the estimated sources ˆ s follows a factorial multivariate Gaussian:
Revised: We assume that the distribution of the estimated sources ˆ s follows a factorial multivariate Gaussian:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Z i  is a constant.
Revised: where Z i > 0 is a constant.

Original: The sufficient statistics θ i, 1 = − µ i σ 

Revised: The sufficient statistics θ i, 1 = − µ i σ

Original: i
Revised: i

Original: and θ i, 2 = 12 σ 2 i are assumed to be linearly independent.
Revised: and θ i, 2 = 12 σ 2 i are assumed to be linearly independent.

Original: We set the variances σ 2 i to be distinct without loss of generality, which can be placed as a constraint during the estimation process as the sufficient statistics of the estimated distribution are trainable.
Revised: We set the variances σ 2 i to be distinct without loss of generality, which can be placed as a constraint during the estimation process as the sufficient statistics of the estimated distribution are trainable.

Original: Note that  the density p ˆ s (ˆ s ) must be a push-forward of p s ( s ) through g − 1 ◦ f , where g − 1 is  the estimated unmixing function. This implicitly restricts p s ( s ) .
Revised: It is noteworthy that regularization on the estimating process is not assumption on the true generating process.

Original: The main result is as follows:
Revised: The main result is as follows:

Original: Theorem 2.1.
Revised: Proposition 2.1.

Revised: Consider the model defined in Eqs.

Original: Assume
Revised: Assume

Original: i. Data points are sampled from a model defined in Eqs. (1-4), where columns of  the Jacobian matrix J f ( s ) are pairwise orthogonal.
Revised: i. (Independent Influences): The influence of each source on the observed variables is independent of each other, i.e., each partial derivative ∂ f /∂s i is independent of the other sources and their influences in a non-statistical sense.

Original: ii.
Revised: ii.

Original: The mixing function f and its inverse are twice differentiable.
Revised: The mixing function f and its inverse are twice differentiable.

Original: iii.
Revised: iii.

Original: The Jacobian determinant of mixing function can be factorized as | det ( J f ( s ))
Revised: The Jacobian determinant of mixing function can be factorized as det( J f ( s ))

Original: | = (cid:81) ni
Revised: = (cid:81) ni

Original: =

Revised: =

Original: y i
Revised: y i

Original: ( s i ) , where y i is an arbitrary function with y i ( s i) > 0 .
Revised: ( s i ) , where y i is a function that depends only on s i .

Original: Note that volume-preserving transformation is a special case when y i ( s i )
Revised: Note that volume-preserving transformation is a special case when y i ( s i )

Original: = 1  .
Revised: = 1 , i = 1 , . . . , n .

Original: Then function h :
Revised: Then function h :

Original: = g− 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Revised: = g ◦ f is a composition of a component-wise invertible transformation and a permutation.

Original: The proof is provided in Appendix A.1.
Revised: The proof is provided in Appendix A.1.

Original: The assumption of pairwise orthogonal columns in the

Original: Jacobian matrix of the mixing function is also formalized as the criterion of IMA by Gresele et al. (2021), which, in other words, assumes that the partial derivatives ∂ f /∂ s i are independent to each other in a non-statistical sense (Gresele et al., 2021) It has been shown theoretically that some spurious solutions can be ruled out by assuming IMA. Specifically, IMA is proved to be useful to distinguish specific models (e.g., conformal maps) from Darmois solutions (Darmois, 1951;
Revised: Note that some part of the proof technique is inspired by or based on Yang et al. (2022).

Original: Hyv ¨ arinen & Pajunen, 1999) and “rotated-Gaussian” measure-preserving automorphisms (MPAs)

Original: (Locatello et al., 2019).

Original: Inspired by this, we adopt Assumption (i) based on the pairwise orthogonality and further show that it is critical to the full identifiability of nonlinear ICA.

Original: The fact that randomly chosen high-dimensional vectors from an isotropic prior are likely to be (near- )orthogonal (Ball et al., 1997) demonstrates that it may be a reasonable assumption in specific (but not rare) scenarios.

Original: To build intuition, one can consider a filmmaking process in a safari park, of which the main characters are wild animals.
Revised: To build intuition, one can consider a filmmaking process in a safari park, of which the main characters are wild animals.

Original: The task of  boom operators here is to record the sound of animals.
Revised: The task of the boom operators here is to record the sound of animals.

Original: As there are many animals and microphones, the mixing (recording) process is strongly dependent on the relative positions between them.
Revised: As there are many animals and microphones, the mixing (recording) process is highly dependent on the relative positions between them.

Original: Thus, the animal (source s i ) influences the recording (mixing function f ) through its position.
Revised: Thus, the animal (source s i ), loosely speaking, influences the recording (mixing function f ) through its position.

Original: The moving direction  can be represented as the  i -th column vector of the Jacobian J ( f ) : ,i .
Revised: The moving direction and speed of them can be loosely interpreted as the partial derivative w.r.t. the i -th source, i.e. ∂ f /∂s i .

Original: Assuming that there is a sufficiently large number of animalsand that they are not cooperative enough to fine-tune their positions  for  better recording, the individual direction
Revised: Assuming that the wild animals are not cooperative enough to fine-tune their positions and speeds for a better recording and the safari park is not crowded, the influences of animals on the recording are generated independently by them in the sense that they are not affected by the others while moving.

Original: J ( f ) : ,i is likely to be almost random. By formalizing that randomness as randomly sampling from a standard Gaussian or other isotropic priors, the vector J ( f ) : ,i that represents its direction is likely to be orthogonal to the others. The connections between the orthogonality and  other criteria have also been discussed by Janzing et al.
Revised: As a result of that generating process, the column vectors of the Jacobian of the mixing function are uncorrelated with each other and the partial derivative w.r.t. to each source is independent of other sources.

Original: ; Gresele et al. (2021) in the literature of causality. Other practical scenarios include fields that adopt  orthogonal coordinate transformations (Gresele et al., 2021), such as dynamic control (Mistry et al., 2010) and structural geology (De Paor, 1983).
Revised: Other practical scenarios include fields that adopt independent influences in process like orthogonal coordinate transformations (Gresele et al., 2021), such as dynamic control (Mistry et al., 2010) and structural geology (De Paor, 1983).

Revised: Besides, similar notion of independent influences may be fundamental to tasks related to representation learning.

Revised: Taking visual representation for instance, the process of generating observable images from latent semantics, e.g., drawing when thinking of specific objects or digital rendering, might be naturally a set of independent mechanisms (Rao & Ruderman, 1998).

Revised: This is somehow consistent with the empirical results in previous works that flow-based generative models (Sorrenson et al., 2020) and variational autoencoders (Khemakhem et al., 2020) are able to reconstruct real-world images pretty well, even when it is not clear whether the generative process satisfies their assumptions on the auxiliary variables (e.g., sufficient variability) because the true real-world generating process is always unknown.

Revised: Thus we believe that the requirement of auxiliary variables might not be necessary in some cases, especially when the true generating process consists of a set of independent mechanisms.

Original: Having said that, a violation of thepairwise orthogonality assumption  in real-world scenarios could be ascribed to a deliberate global adjustment of sources.
Revised: Having said that, a violation of the assumption of independent influences in real-world scenarios could be ascribed to a deliberate global adjustment of sources.

Original: For example, cinema audio systems are carefully adjusted to achieve a homogeneous sound effect on every audience.
Revised: For example, cinema audio systems are carefully adjusted to achieve a homogeneous sound effect on every audience.

Original: The position and orientation of each speaker are fine-tuned according to the others.
Revised: The position and orientation of each speaker are fine-tuned according to the others.

Original: In this case, it may be difficult to distinguish the influences from different speakers because of the fine-tuning.
Revised: In this case, it may be difficult to distinguish the influences from different speakers because of the fine-tuning.

Original: This leads to a high degree of multicollinearity across the columns of Jacobian, thus violating the assumption of pairwise orthogonality.
Revised: This may lead to a high degree of multicollinearity across the columns of Jacobian, thus violating the assumption of independent influences.

Original: Other possible violations can also be found in similar scenarios where the influences from different sources are fine-tuned in order to achieve homogeneity (e.g., the cocktail party problem described by Gresele et al.
Revised: Other possible violations can also be found in similar scenarios where the influences from different sources are fine-tuned in order to achieve homogeneity (e.g., the cocktail party problem described by Gresele et al.

Original: (2021)), or in an unlikely scenario where the mixing process is naturally component-wise (e.g., an autoregressive model (Papamakarios et al., 2021)).
Revised: (2021)), or in an unlikely scenario where the mixing process is naturally component-wise (e.g., an autoregressive model (Papamakarios et al., 2021)).

Revised: Future works include validating the proposed conditions in complicated real-world tasks.

Original: Regarding Assumption (iii), we first note that mixing functions with factorial Jacobian determinants are of a much wider range as compared to component-wise transformations.
Revised: Regarding Assumption (iii), we first note that mixing functions with factorial Jacobian determinants are of a much wider range as compared to component-wise transformations.

Original: To see this, a straightforward example is the volume-preserving transformation, which has been widely adopted in generative models like those based on flows (Zhang et al., 2021).
Revised: To see this, a straightforward example is the volume-preserving transformation, which has been widely adopted in generative models like those based on flows (Zhang et al., 2021).

Original: In fact, all transformations with constant Jacobian determinant can be factorized w.r.t.
Revised: In fact, all transformations with constant Jacobian determinant can be factorized w.r.t.

Original: the sources, which, generally speaking, are not component-wise.
Revised: the sources, which, generally speaking, are not component-wise.

Original: Besides, there exists other non-volume-preserving transformations with factorial Jacobian determinant. The empirical results in Yang et al.
Revised: Besides, there exists other non-volume-preserving transformations with factorial Jacobian determinant, 1 as demonstrated by the empirical results in Yang et al.

Original: (2022) indicate this, though our current knowledge of this type of function is rather limited.
Revised: (2022), though our current knowledge of this type of function is rather limited.

Original: Moreover, the volume-preserving assumptionUnder at 2 has been demonstrated by Sorrenson et al.
Revised: Moreover, the volume-preserving assumption has been demonstrated by Sorrenson et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2022) to be helpful to weaken the requirement of auxiliary variables, specifically by reducing the number of required labels.
Revised: (2022) to be helpful to weaken the requirement of auxiliary variables, specifically by reducing the number of required labels.

Original: Our work takes this further by imposing a weaker version, i.e., Assumption (iii),  together with  other constraints,which completely removes the need for any auxiliary variable.
Revised: Our work takes this further by imposing a weaker version, i.e., Assumption (iii), which, together with the other constraints, removes the need for any auxiliary variable.

Original: Theorem 2.1 opens up the possibility of thefull identifiability of nonlinear ICA while maintaining the standard mutually marginal independence assumption of sources.
Revised: Proposition 2.1 demonstrates the possibility of the identifiability of nonlinear ICA while maintaining the standard mutually marginal independence assumption of sources.

Original: This is consistent with the original setting of ICA and plays an important role in a more general range of unsupervised learning tasks, compared to previous works relying on conditional independence given auxiliary variables.
Revised: This is consistent with the original setting of ICA and plays an important role in a more general range of unsupervised learning tasks, compared to previous works relying on conditional independence given auxiliary variables.

Original: In fact, one can easily modify Theorem 2.1 to include generating process based on auxiliary variables by imposing similar restrictions on the mixing process.
Revised: In fact, one may modify Proposition 2.1 to consider the generating process based on auxiliary variables by imposing similar restrictions on the mixing process.

Original: Different from previous works (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019; Lachapelle et al., 2021; Yang et al., 2022), this extension removes common restrictions on the required auxiliary variables, such as the number of distinct values (e.g., labels), non-proportional variances, or sufficient variability.
Revised: Different from previous works (Hyv¨arinen & Morioka, 2016; 2017; Hyv¨arinen et al., 2019; Lachapelle et al., 2022; Yang et al., 2022), this extension removes common restrictions on the required auxiliary variables, such as the number of distinct values (e.g., labels), non-proportional variances, or sufficient variability.

Original: Therefore, although our main result (Theorem 2.1) focuses on nonlinear ICA with unconditional priors, italso provides potential insight on improving the flexibility of utilizing the auxiliary variable when it is available.
Revised: Therefore, although our main result (Proposition 2.1) focuses on nonlinear ICA with unconditional priors, it provides potential insight into improving the flexibility of utilizing the auxiliary variable when it is available.

Original: As a trade-off, the additional assumptions on the mixing process might limit its usage in some scenarios.
Revised: As a trade-off, the additional assumptions on the mixing process might limit its usage.

Original: 2. I DENTIFIABILITY UP TO COMPONENT - WISE LINEAR TRANSFORMATION
Revised: 2. I DENTIFIABILITY UP TO COMPONENT - WISE LINEAR TRANSFORMATION

Original: Most works in nonlinear ICA focus on identifiability up to a component-wise invertible transformation and a permutation (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; H¨alv¨a & Hyvarinen, 2020; Khemakhem et al., 2020), which is also called permutation-identifiability in the literature of disentanglement (Lachapelle et al., 2021).
Revised: Most works in nonlinear ICA focus on identifiability up to a component-wise invertible transformation and a permutation (Hyv ¨ arinen & Morioka, 2016; Hyv ¨ arinen et al., 2019; H ¨ alv ¨ a & Hyv ¨ arinen, 2020; Khemakhem et al., 2020), which is also called permutation-identifiability in the literature of disentanglement (Lachapelle et al., 2022).

Original: This indeterminacy is trivial compared to the fundamental nonuniqueness of nonlinear ICA (Hyv ¨ arinen & Pajunen, 1999) and is analogous to the indeterminacy involving rescaling and permutation in linear ICA (Taleb & Jutten, 1999).
Revised: This indeterminacy is trivial compared to the fundamental nonuniqueness of nonlinear ICA (Hyv ¨ arinen & Pajunen, 1999) and is analogous to the indeterminacy involving rescaling and permutation in linear ICA (Hyv ¨ arinen & Pajunen, 1999).

Original: Recently, Yang et al.
Revised: Recently, Yang et al.

Original: (2022) provided an identifiability result based on auxiliary variables that further reduces the indeterminacy of the component-wise nonlinear transformation to a linear one.
Revised: (2022) provided an identifiability result based on auxiliary variables that further reduces the indeterminacy of the component-wise nonlinear transformation to a linear one.

Original: Inspired by it, the following theorem provides  identifiability conditions of nonlinear ICA up to a component-wise linear transformation with unconditional priors.
Revised: Inspired by it, the following result provides the identifiability conditions of nonlinear ICA up to a component-wise linear transformation with unconditional priors.

Original: This is interesting as it opens up the possibility forpotential further optimization with only element-wise linear operations, such as the Hadamard product.
Revised: This reduced indeterminacy could give rise to a more informative disentanglement and opens up the possibility for further improving the quality of recovery with only element-wise linear operations, such as the Hadamard product.

Revised: The result is formally stated as follows, with a proof provided in Appendix A.2.

Original: Theorem 2.2.
Revised: Proposition 2.2.

Original: Besides the assumptions in Theorem 2.1, assume that the source distribution p s ( s ) is a factorial multivariate Gaussian and function f is volume-preserving.
Revised: Besides the assumptions in Proposition 2.1, assume that the source distribution p s ( s ) is a factorial multivariate Gaussian and function f is volume-preserving.

Original: Then h := g− 1 ◦ f is a composition of a component-wise linear transformation and a permutation.
Revised: Then h := g ◦ f is a composition of a component-wise linear transformation and a permutation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Setup.
Revised: Setup.

Original: Based on the proposed theorems, we use a volume-preserving flow to recover the sources up to the indeterminacies mentioned above.
Revised: Based on the proposed results, we use a volume-preserving flow to recover the sources up to the indeterminacies mentioned in Section 2.1.

Original: Similar to (Gresele et al., 2021), we consider a regularized maximum likelihood estimation to enforce orthogonality between columns of the Jacobian.
Revised: Similar to (Gresele et al., 2021), we consider a regularized maximum likelihood estimation to enforce orthogonality between columns of the Jacobian.

Original: Differently, we propose a new objective to unify the norm of columns in the estimation, i.e., L ( g ; x) = E x (cid:34) log [Equation]
Revised: Differently, we propose an objective to unify the norm of columns in the estimation, which, with a slight abuse of notation, is as follows:

Original: ( x )

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: (cid:88) i =

Original: [Equation]

Original: (cid:13)

Original: [Equation]

Original: (cid:13)(cid:13) ∂g

Original: [Equation]

Original: [Equation]

Original: i

Original: [Equation] − [Equation] (cid:33)(cid:35) , where λ 1 is a regularization parameter.
Revised: where λ 1 is a regularization parameter and ∥ · ∥ 2 denotes the Euclidean norm.

Original: The following proposition summarizes its related properties:
Revised: The following proposition summarizes its related properties:

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: The following inequality holds
Revised: The following inequality holds

Original: [Table]
Revised: [Equation]

Original: with equality iff.

Original: J g − 1 (ˆ s )

Original: =

Original: O (ˆ s ) λ

Original: (ˆ s ) , where O (ˆ s ) is an orthogonal matrix and λ (ˆ s ) is a scalar.

Original: The proof is provided in Appendix 3.1.

Original: We train a volume-preserving flow to maximize L .

Original: We first conduct an ablation study to verify the necessity of the proposed assumptions.

Original: [Figure]
Revised: [Figure]

Original: 4 R e c o v e r e d
Revised: 4 R e c o v e r e d

Original: S o u
Revised: S o u

Original: r c e s
Revised: r c e s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Full .
Revised: Full .

Original: 4 Ground
Revised: 4 Ground

Original: [Figure]
Revised: [Figure]

Original: 4 R e c o v e r e d
Revised: 4 R e c o v e r e d

Original: S o u
Revised: S o u

Original: r c e s
Revised: r c e s

Original: Correlation Matrix 0.
Revised: Correlation Matrix 0.

Original: (b) VP .
Revised: (b) VP .

Original: 4 Ground
Revised: 4 Ground-

Original: [Figure]
Revised: [Figure]

Original:     4 R e c o v e r e d
Revised: Sources    4 R e c o v e r e d

Original: S o u
Revised: S o u

Original: r c e s
Revised: r c e s

Original: Correlation Matrix 0.
Revised: Correlation Matrix 0.

Original: (c
Revised: (c) NVP .

Revised: with equality iff.

Original: [Figure]
Revised: J g ( x ) =

Original: NVP .
Revised: O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar. The proof is provided in Appendix 3.1. We train a volume-preserving flow to maximize L ( g ; x ) .

Original: [Figure]
Revised: [Figure]

Revised: We conduct an ablation study to verify the necessity of the proposed assumptions.

Original: Specifically, we focus on the two major conditions, i.e., orthogonal columns and volume-preserving.
Revised: Specifically, we focus on the two major conditions, i.e., independent influences and factorial change of the volume.

Original: For volume-preserving transformations, we train a volume-preserving flow to maximize the likelihood, which constraints the Jacobian determinant of GLOW (Kingma & Dhariwal (2018)).
Revised: We train a volume-preserving flow to maximize the likelihood, which constrains the Jacobian determinant of GLOW (Kingma & Dhariwal, 2018).

Original: For nonvolume-preserving transformations, we use  GLOW instead.
Revised: For non-volumepreserving transformations, we use the original version of GLOW instead.

Original: The observational data are generated according to the required assumptions, as detailed in Appendix B.
Revised: The observational data are generated according to the required assumptions, as detailed in Appendix B.

Original: We evaluate the models by the mean correlation coefficient (MCC) between ground truth and reconstructed sources.

Original: Results.
Revised: Results.

Original: Results for each model are summarised in Figure 1.
Revised: Results for each model are summarised in Figure 1.

Original: We construct each model as follows: (Full) All conditions satisfied; (VP) Compared to Full , columns of Jacobian of (un)mixing functions are not orthogonal; (NVP) Compared to VP , (un)mixing functions are not volume-preserving.
Revised: We construct each model as follows: (Full) Influences are mutually independent and transformations are volume-preserving; (VP) Compared to Full , the assumption of independent influences is violated; (NVP) Compared to VP , (un)mixing functions are not volume-preserving.

Original: One can observe that when the proposed assumptions are fully satisfied ( Full ), our model achieves the highest MCC on average.
Revised: One observes that when the proposed assumptions are fully satisfied ( Full ), our model achieves the highest mean correlation coefficient (MCC) on average.

Original: Besides, according to the decline of performance after violating each of these conditions, the necessity of them is supported.
Revised: Besides, according to the decline of performance after violating each of these conditions, the necessity of them appears to be supported.

Original: The visualization of the Pearson correlation matrices is shown in Figure 2.
Revised: The visualization of the Pearson correlation matrices is shown in Figure 2.

Original: All results are of 10 trials.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We provide the full identifiability result for nonlinear ICA with unconditional priors.
Revised: We provide an identifiability result for nonlinear ICA with unconditional priors.

Original: In particular, we prove that the i.i.d.
Revised: In particular, we prove that the i.i.d.

Original: latent sources can be recovered up to a component-wise invertible transformation and a permutation with only conditions on the nonlinear mixing process.
Revised: latent sources, under certain assumptions, can be recovered up to a componentwise invertible transformation and a permutation with only conditions on the nonlinear mixing process.

Original: To the best of our knowledge, this is the first identifiability result of nonlinear ICA with unconditional priors while not restricting it to the two-dimensional or post-nonlinear cases.

Original: Therefore it is consistent with the original setting of ICA that is based on thestandard marginal independence assumption of latent sources, while previous works rely on conditional independence on auxiliary variables as weak supervision or inductive bias.
Revised: Therefore it may stay closer to the original notion of ICA that is based on the marginal independence assumption of latent sources, while previous works rely on conditional independence on auxiliary variables as weak supervision or inductive bias.

Original: Besides, by further assuming the Gaussianity of sources, we prove that the indeterminacy of the component-wise transformation can be reduced to a component-wise linear transformation, which sheds light on potential further optimizations by element-wise linear operations.
Revised: Besides, by further assuming the Gaussianity of sources, we prove that the indeterminacy of the component-wise transformation can be reduced to a component-wise linear transformation, which sheds light on potential further optimizations by element-wise linear operations.

Original: By including the auxiliary variableas a special case but without common constraints on it (e.g., number of class labels), our results may provideextra flexibility in specific scenarios when auxiliary variables are available.
Revised: By including the auxiliary variable but without common constraints on it, our results may provide flexibility in specific scenarios when auxiliary variables are available.

Original: Limitation and discussion.
Revised: Limitation and discussion.

Original: Although our identifiability resultis in some sense more general than previous works as it removes the need for auxiliary variables, the assumptions on the mixing process may restrict its usage in specific scenarios as discussed in Section 2.1.
Revised: Although our identifiability result removes the need for auxiliary variables, the assumptions on the mixing process may restrict its usage in specific scenarios as discussed in Section 2.1.

Original: This is inevitably a trade-off between introducing auxiliary variables and imposing restrictions on the mixing process in order to achieve full identifiability, whose practical usage depends on the scenario and information available.
Revised: This is inevitably a trade-off between introducing auxiliary variables and imposing restrictions on the mixing process in order to achieve the identifiability, whose practical usage depends on the scenario and information available.

Original: It is worth noting that Gresele et al.
Revised: It is worth noting that Gresele et al.

Original: (2021) assumed orthogonality, conformal map, and the others, which are already rather restrictive, in order to rule out two specific types of spurious solutions (i.e., 
Revised: (2021) assumed orthogonality, conformal map and the others in order to rule out two specific types of spurious solutions (i.e., Darmois construction (Darmois, 1951) and “rotated-Gaussian” MPAs (Locatello et al., 2019)) without introducing auxiliary variables. Our future work includes validating the proposed conditions in complicated real-world tasks and generalizing some of them. A CKNOWLEDGEMENTS

Original: Darmois construction and “rotated-Gaussian” MPAs) without introducing auxiliary variables.
Revised: We thank the anonymous reviewers for their constructive comments and helpful criticisms. This work was supported in part by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by the United States Air Force under Contract No. FA8650-17-C and by a grant from Apple. The NIH or NSF is not responsible for the views reported in this paper

Original: A P ROOFS A. P ROOF OF T HEOREM 2. 
Revised: A P ROOFS A. P ROOF OF P ROPOSITION 2.

Original: Theorem 2.1.
Revised: We restate the setup here. Consider the following data-generating process of nonlinear ICA p

Original: Assume
Revised: [Equation]

Original: i. Data points are sampled from a model defined in Eqs. (1-4), where columns of the Jacobian matrix J f ( s ) are pairwise orthogonal. ii.
Revised: where x denotes the observed random vector, s is the latent random vector representing the marginally independent sources, p s i is the marginal probability density function (pdf) of s i ,

Original: The mixing function f and its inverse are twice differentiable. iii.
Revised: p s is the joint pdf, and f : s → x denotes a nonlinear mixing function.

Original: The Jacobiandeterminant of  mixing function can be factorized as | det ( J f ( s ))
Revised: We preprocess the observational data by centering the columns of the Jacobian of the mixing function.

Original: | = (cid:81) ni
Revised: This could be achieved by applying a fixed transformation on the observational data that is equivalent to left multiplying the Jacobian of the mixing function with the centering matrix U :

Original: [-=
-]
Revised: [-=
-]=

Original: y i ( s i )  , where y i is an arbitrary function with y i
Revised: I n − (1 /n ) 11 ⊤ , where I n is the identity matrix of size n and 1 is the n -dimensional vector of all ones.

Original: ( s i ) > 0 .
Revised: After centering, according to the assumption of independent influences, the columns of the Jacobian matrix are uncorrelated and of zero-means, and thus orthogonal to each other 2 .

Original: Note that volume-preserving transformation  is a special case when y i ( s i )
Revised: Since the preprocessing transformation U is fixed w.r.t. no variables other than n , which is the fixed number of dimensions, it could always be recovered after reconstruction. We assume that the distribution of the estimated sources ˆ s follows a factorial multivariate Gaussian:

Original: = 1 .
Revised: [Equation]

Original: Then function h : = g − 1 ◦ f is a composition of{+constant. The sufficient statistics θ i, 1 = − µ i σ 
+}
Revised: where Z i > 0 is a {+constant. The sufficient statistics θ i, 1 = − µ i σ 
+}

Original: a
Revised: i

Original: component-wise invertible transformation and  a permutation. Proof.
Revised: and θ i, 2 = 12 σ 2 i are assumed to be linearly independent. We set the variances σ 2 i to be distinct without loss of generality, which can be placed as a constraint during the estimation process as the sufficient statistics of the estimated distribution are trainable.

Original: Our goal here is to show that function h  is a  permutation with a component-wise invertible transformation of sources.
Revised: Our goal here is to show that function h = g ◦ f is a composition of a permutation and a componentwise invertible transformation of sources.

Original: By using chain rule repeatedly, we write the Jacobian of function h :=
Revised: By using chain rule repeatedly, we write the Jacobian of function h :=

Original: g− 1 ◦ fequivalently as
Revised: g ◦ f as

Original: [Equation]
Revised: [Equation]

Original: where ˆ λ ( x ) is a scalar,
Revised: where ˆ λ ( x ) is a scalar,

Original: ˆ O ( x ) and O ( s ) represent the corresponding orthogonal matrices, and D ( s ) denotes the corresponding diagonal matrix.
Revised: ˆ O ( x ) and O ( s ) represent the corresponding orthogonal matrices, and D ( s ) denotes the corresponding diagonal matrix.

Original: The fourth equality follows from assumption i.
Revised: The fourth equality follows from assumption i.

Original: Since J h ( s ) =
Revised: Since J h ( s ) =

Original: ˆ λ ( x ) ˆ O
Revised: ˆ λ ( x ) ˆ O

Original: ( x ) O ( s ) D ( s ) by Eq.
Revised: ( x ) O ( s ) D ( s ) by Eq.

Original: 5 and ˆ O ( x ) O ( s ) is an orthogonal matrix, the columns of J h ( s ) are orthogonal to each other, i.e.,[Equation]
Revised: 4 and ˆ O ( x ) O ( s ) is an orthogonal matrix, the columns of J h ( s ) are orthogonal to each other, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where J h ( s ) : ,j is the j -th column ofJacobian matrix J h ( s ) .
Revised: where J h ( s ) : ,j is the j -th column of matrix J h ( s ) .

Original: Then it follows that
Revised: This can be rewritten as n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By assumption, the components of the true sources s are mutually independent, while the estimated sources ˆ s follows a multivariate Gaussian distribution:
Revised: By assumption, the components of the true sources s are mutually independent, while the estimated sources ˆ s follows a multivariate Gaussian distribution:

Original: [Equation]
Revised: [Equation]

Original: where Z i  is a constant.
Revised: where Z i > 0 is a constant.

Original: The sufficient statistics θ i, 1 and θ i, 2 are assumed to be linearly independent.
Revised: The sufficient statistics θ i, 1 and θ i, 2 are assumed to be linearly independent.

Original: Applying the change of variable rule, we have p s ( s ) =
Revised: Applying the change of variable rule, we have p s ( s ) =

Original: p ˆ s (ˆ s ) | det( J h ( s ))
Revised: p ˆ s (ˆ s ) | det( J h ( s ))

Original: | . According to Eq. 8, we have the following equation after taking the logarithm on both sides:
Revised: | , which, by plugging in Eq. 6 and taking the logarithm on both sides, yields

Original: [Equation]
Revised: [Equation]

Original: According to assumption iii, one can factorize | det ( J h ( s )) | as (cid:81) ni
Revised: According to assumption iii, we have the factorization det( J h ( s )) = (cid:81) ni

Original: =

Revised: =

Original: y i
Revised: y i

Original: ( s i ) , where y i is an arbitrary function that y i ( s i) > 0 .[-
-]
Revised: ( s i ) , where y i is a function that depends only on s i .[-
-]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: We use second-order Taylor polynomial to expand h i as h i is assumed to be twice-differentiable:
Revised: Taking second-order derivatives of both sides of Eq. 8 w.r.t. s j and s k with j ̸ = k , we obtain

Original: [Equation]
Revised: [Equation]

Original: Same as the proof of Lemma 1 from Equation 17 toEquation 18 in (Yang et al., 2022), taking the Taylor expansion of  h i in Eq.
Revised: According to the assumption of independent influences as well as corresponding restriction on the estimating process, for all j ̸ = k , we have ∂ 2 h i ∂s j ∂s k = 0 .

Original: 9, we have n{+Since θ i, 
+}
Revised: {+Since θ i, 
+}

Original: [Equation] where C i is a constant. Because RHS has cross terms { ∆s j ∆s k } j ̸ = k while LHS does not, the coefficients of them are zeros:
Revised: and θ i, 2 are sufficient statistics of a multivariate Gaussian, we have θ i, 2 = 12 σ 2 i , where σ i represents the standard deviation. Plugging it into Eq.

Revised: 9 yields

Original: [Equation]
Revised: [Equation]

Original: Note that the derivation of Eq.
Revised: Note that a similar formulation of Eq.

Original: 13 is directly based on that of Lemma 1 in (Yang et al., 2022).
Revised: 10 is also derived and presented in Lemma 1 by Yang et al.

Original: The only difference is the term y i
Revised: The only difference is the term y i

Original: ( s i ) , which does not affect the process as it has no cross terms.
Revised: ( s i ) , which does not affect the formulation as it has no cross terms.

Original: Because θ i, 1 and θ i, 2 are sufficient statistics of a multivariate Gaussian, then θ i, 1 = − µ i σ 2 i and θ i, 2 = 12 σ 2 i , where µ i and σ i represent mean and standard deviation, respectively.
Revised: Thus, based on Lemma 1 in (Yang et al., 2022), we can also obtain Eq. 10 with the current assumptions or alternatively weaker ones.

Original: We set θ i, 1 = 0 . Plugging it into Eq.
Revised: Then, we define the following matrices: [Equation]

Original: 13 yields
Revised: Σ (

Original: [Equation]
Revised: [Equation]

Original: Define the following matrices:
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Revised: (cid:88)

Revised: i

Revised: [Equation]

Revised: (cid:18)

Revised: ∂h

Revised: [Equation]

Revised: ∂s

Revised: [Equation]

Revised: (cid:19)

Revised: [Equation]

Revised: n (cid:88)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: (cid:18) ∂h i ∂s

Revised: [Equation]

Revised: (cid:33)

Revised: [Equation]

Revised: Σ (

Revised: [Equation]

Revised: s )

Revised: = diag (cid:32) n (cid:88) i = (cid:18)

Revised: ∂h

Revised: i ∂s(cid:19), · · · , n (cid:88)

Revised: [Equation]

Revised: [Equation]

Original: Then we rewrite Eq.
Revised: We rewrite Eq.

Original: 14 and Eq.
Revised: 10 and Eq.

Original: 7 as
Revised: 5 as

Original: [Equation]
Revised: [Equation]

Original: Because all elements in matrices defined in Eq.
Revised: Because all elements in matrices defined in Eq.

Original: 15 are positive, we can take sqaure roots on both sides of equations in Eq.
Revised: 11 are positive, we can take sqaure roots on both sides of equations in Eq.

Original: 16, yielding
Revised: 12, yielding

Original: [Equation]
Revised: [Equation]

Original: where O 1 and O 2 are the corresponding orthogonal matrices.
Revised: where O 1 and O 2 are the corresponding orthogonal matrices ( O T 1 O 1 =

Revised: O T 2 O 2 = I n ) and Λ 1 /denotes the matrix by applying entry-wise square root to the entries of diagonal matrix Λ .

Original: Then we rewrite Eq.
Revised: Then we rewrite Eq.

Original: 17 as
Revised: 13 as

Original: [Equation]
Revised: [Equation]

Original: By reformulating, 
Revised: By reformulating, we have

Original: [Equation]
Revised: [Equation]

Original: Inspired by Yang et al.

Original: , we consider both sides of Eq.
Revised: We consider both sides of Eq.

Original: 19 as a singular value decomposition of the LHS.
Revised: 14 as a singular value decomposition of the LHS.

Original: Because σ j ̸ = σ k , ∀ j ̸ = k , elements in Λ 1 / 2 1 Λ − 1 / 2are distinct, as well as non-degenerate and non-zero.
Revised: Since σ j ̸ = σ k , ∀ j ̸ = k , elements in Λ 1 / 2 1 Λ − 1 / 2are distinct. Therefore, non-zero elements in Λ 1 / 2 1 Λ − 1 / 2are also distinct because Λ 2 is an identity matrix. Because the mixing function is assumed to be invertible, the Jacobian of the mixing function J f ( s ) = O ( s ) D ( s ) is of full-rank.

Revised: Therefore, all diagonal elements in the diagonal matrix Λ 1 / 2 1 Λ − 1 / 2are non-zero.

Original: Thus, based on the uniqueness of singular value decomposition (see e.g. Theorem 4.1 in Trefethen & Bau III (1997)), O 2 is a composition of a permutation matrix and a signature matrix.
Revised: Then, based on the uniqueness of singular value decomposition (see e.g. Proposition 4.1 in Trefethen & Bau III (1997)), O 2 is a composition of a permutation matrix and a signature matrix.

Original: Therefore, J h ( s ) = Λ − 1 / 

Revised: Therefore, J h ( s ) = Λ − 1 /

Original: O 2 Σ(2 , s ) 1 / 2 is a generalized permutation matrix.
Revised: O 2 Σ (2 , s ) 1 / 2 is a generalized permutation matrix.

Original: It thus follows that the function h :=
Revised: It thus follows that the function h :=

Original: g− 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Revised: g ◦ f is a composition of a component-wise invertible transformation and a permutation.

Original: A. P ROOF OF T HEOREM 2. 
Revised: A. P ROOF OF P ROPOSITION 2.

Original: Theorem 2.2.
Revised: Proposition 2.2.

Original: Besides the assumptions in Theorem 2.1, assume that the source distribution p s ( s ) is a factorial multivariate Gaussian and function f is volume-preserving.
Revised: Besides the assumptions in Proposition 2.1, assume that the source distribution p s ( s ) is a factorial multivariate Gaussian and function f is volume-preserving.

Original: Then h := g− 1 ◦ f is a composition of a component-wise linear transformation and a permutation.
Revised: Then h := g ◦ f is a composition of a component-wise linear transformation and a permutation.

Original: Proof.
Revised: Proof.

Original: Since s is from a factorial multivariate Gaussian distribution, we represent its density as
Revised: Since s is from a factorial multivariate Gaussian distribution, we represent its density as

Original: [Equation]
Revised: [Equation]

Original: According to Theorem 2.1, function h is a component-wise invertible transformation of sources, i [Equation] e.,
Revised: According to Proposition 2.1, function h is a component-wise invertible transformation of sources, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two equations with Eq.
Revised: Combining the above two equations with Eq.

Original: 9 and the condition that h is volume-preserving, it follows that
Revised: 7 and the condition that h is volume-preserving, it follows that

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Clearly, h i ( s i ) is a linear function of s i .
Revised: Therefore, h i ( s i ) is a linear function of s i .

Original: A. P ROOF OF P ROPOSITION 3. 
Revised: A. P ROOF OF P ROPOSITION 3.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: The following inequality holds
Revised: The following inequality holds

Original: [Equation]
Revised: [Equation]

Original: with equality iff.
Revised: with equality iff.

Original: J g − 1 (ˆ s ) =
Revised: J g ( x ) =

Original: O (ˆ s ) λ (ˆ s ) [Equation] where O (ˆ s ) is an orthogonal matrix and λ (ˆ s ) is a scalar.
Revised: O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar.

Original: Proof.
Revised: Proof.

Original: According to Hadamard’s inequality [Equation] we have
Revised: According to Hadamard’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: with equality iff.
Revised: with equality iff.

Original: vectors∂g − 1 ∂ 
Revised: vectors ∂ g ∂x

Original: ˆ s i (ˆ s ) are orthogonal.
Revised: i , i = 1 , 2 , . . . , n are orthogonal.

Original: Then applying the inequality of arithmetic and geometric means yields
Revised: Then applying the inequality of arithmetic and geometric means yields

Original: [Equation]
Revised: [Equation]

Original: with the first equality iff.
Revised: with the first equality iff.

Original: for all i in { 1 , 2 , · · · , n} , (cid:13)(cid:13)(cid:13)∂g − 1 ∂ 
Revised: for all i = 1 , 2 , . . . , n , (cid:13)(cid:13)(cid:13) ∂ g ∂x i [Equation]

Original: ˆ s i (cid:13)(cid:13)(cid:13) is equal.
Revised: 2 is equal.

Original: Because J g − 1 (ˆ s ) is non-singular,∂g − 1 ∂ ˆ s i are orthogonal and equal to each other for all i in { 1 , 2 , · · · , n} iff.
Revised: Because J g ( x ) is non-singular, ∂ g ∂x i are orthogonal and equal to each other for all i = 1 , 2 , . . . , n iff.

Original: J g − 1 =
Revised: J g ( x ) =

Original: O (ˆ s ) λ (ˆ s ) .
Revised: O ( x ) λ ( x ) .

Original: B E XPERIMENTS B. S YNTHETIC DATASETS AND IMPLEMENTATION DETAILS
Revised: B E XPERIMENTS B. S YNTHETIC DATASETS AND IMPLEMENTATION DETAILS

Original: In order to generate observational data satisfying required assumptions, we synthetic the sources and mixing process as follows:
Revised: In order to generate observational data satisfying required assumptions, we simulate the sources and mixing process as follows:

Original: Full .
Revised: Full .

Original: Here we describe the generating process that satisfies all our proposed assumptions.

Original: Following Gresele et al.
Revised: Following Gresele et al.

Original: (2021), we generate the mixing functions based on M¨obius transformations with scaled sources and constant volume.
Revised: (2021), we generate the mixing functions based on M ¨ obius transformations with scaled sources and constant volume.

Original: According to Liouville’s results (Flanders (1966), also summarized in Theorem F.2 in Gresele et al.
Revised: According to Liouville’s results (Flanders, 1966) (also summarized in Theorem F.2 in Gresele et al.

Original: (2021)), the M¨obius transformation guarantees the orthogonality between the columns vectors of its Jacobian matrix, which achieves uncorrelatedness after centering.
Revised: , the M ¨ obius transformation guarantees the orthogonality between the columns vectors of its Jacobian matrix, which achieves uncorrelatedness after centering.

Original: Different from Gresele et al.
Revised: Different from Gresele et al.

Original: (2021), we scaled the sources while preserving volumes before M¨obius transformation with distinct scalers to make sure that the generating process is not a conformal map.
Revised: (2021), we scaled the sources while preserving volumes before M¨obius transformation with distinct scalers to make sure that the generating process is not a conformal map.

Original: VP .
Revised: VP .

Original: Here we describe the generating process that is volume-preserving but not necessarily with orthogonal columns of Jacobian.
Revised: Here we describe the generating process that is volume-preserving but not necessarily with orthogonal columns of Jacobian.

Original: We use volume-preserving flow called GIN (Sorrenson et al. to generate the mixing function.
Revised: We use volume-preserving flow called GIN (Sorrenson et al., 2020) to generate the mixing function.

Original: GIN is a volume-preserving version of RealNVP (Dinh et al. (2016)), which achieves volume preservation by setting the scaling function of the final component to the negative sum of previous ones.
Revised: GIN is a volume-preserving version of RealNVP (Dinh et al., 2016), which achieves volume preservation by setting the scaling function of the final component to the negative sum of previous ones.

Original: 4 We use the official implementation of GIN (Sorrenson et al. (2020)), which isalso part of FrEIA.
Revised: 3 We use the official implementation of GIN (Sorrenson et al., 2020), which is part of FrEIA.

Original: NVP .
Revised: NVP .

Original: Here we describe the generating process without assumptions on volume preservation and orthogonal columns of Jacobian.
Revised: Here we describe the generating process without assumptions on volume preservation and orthogonal columns of Jacobian.

Original: Following Sorrenson et al. (2020), we use GLOW (Kingma &
Revised: Following (Sorrenson et al., 2020), we use GLOW (Kingma &

Original: Dhariwal, 2018) to generate the mixing function.
Revised: Dhariwal, 2018) to generate the mixing function.

Original: The difference between the coupling block in
Revised: The difference between the coupling block in

Original: GLOW and GIN is that the Jacobian determinant of the former is not constrained to be one.
Revised: GLOW and GIN is that the Jacobian determinant of the former is not constrained to be one.

Original: The implementation of GLOW is also included in the official implementation of GIN (Sorrenson et al. (2020)), which is also part of FrEIA.
Revised: The implementation of GLOW is also included in the official implementation of GIN (Sorrenson et al., 2020), which is also part of FrEIA.

Original: Regarding the model evaluation, we use the mean correlation coefficient (MCC) between the groundtruth and recovered latent sources.
Revised: Regarding the model evaluation, we use the mean correlation coefficient (MCC) between the groundtruth and recovered latent sources.

Original: We first compute pair-wise correlation coefficients between the true sources and recovered ones.
Revised: We first compute pair-wise correlation coefficients between the true sources and recovered ones.

Original: Then we solve a linear sum assignment problem to match each recovered source to the ground truth with the highest correlation between them.
Revised: Then we solve a linear sum assignment problem to match each recovered source to the ground truth with the highest correlation between them.

Original: MCC is a standard metric to measure the degree of identifiability up to component-wise transformation in the literature (Hyvarinen & Morioka, 2016).
Revised: MCC is a standard metric to measure the degree of identifiability up to component-wise transformation in the literature (Hyv¨arinen & Morioka, 2016).

Original: Following previous works (Khemakhem et al. , the ground-truth sources ( n = 5 ) are sampled from a multivariate Gaussian, with zero means and variances sampled from a uniform distribution on [ . 5 , 3] 5 .
Revised: Following previous works (Khemakhem et al., 2020), the ground-truth sources ( n = 5 ) are sampled from a multivariate Gaussian, with zero means and variances sampled from a uniform distribution on [0 . 5 , 3] 4 .

Original: The sample size for all experiments is 10000.
Revised: The sample size for all experiments is 10000 .

Original: The learning rate is 0 .
Revised: The learning rate is 0 .

Original: 01 and batch size is 1000 .
Revised: 01 and batch size is 1000 .

Original: The number of coupling layers for both GIN and GLOW is set as 24.
Revised: The number of coupling layers for both GIN and GLOW is set as 24 .

Original: The estimating model is built in the reverse direction as described above.
Revised: The estimating model is built in the reverse direction as described above.

Original: The code of our experiments is in Python and developed based on GIN (Sorrenson et al.
Revised: The code of our experiments is in Python and developed based on GIN (Sorrenson et al., 2020).