On the Identifiability of Nonlinear ICA: Sparsity and Beyond
Abstract
Nonlinear independent component analysis (ICA) aims to recover the underlying independent latent sources from their observable nonlinear mixtures.
How to make the nonlinear ICA model identifiable up to certain trivial indeterminacies is a long-standing problem in unsupervised learning.
Recent breakthroughs reformulate the standard independence assumption of sources as conditional independence given some auxiliary variables (e.g., class labels and/or domain/time indexes) as weak supervision or inductive bias.
However, nonlinear ICA with unconditional priors cannot benefit from such developments.
We explore an alternative path and consider only assumptions on the mixing process, such as Structural Sparsity .
We show that under specific instantiations of such constraints, the independent latent sources can be identified from their nonlinear mixtures up to a permutation and a component-wise transformation, thus achieving nontrivial identifiability of nonlinear ICA without auxiliary variables.
We provide estimation methods and validate the theoretical results experimentally.
The results on image data suggest that our conditions may hold in a number of practical data generating processes.
Introduction
Nonlinear independent component analysis (ICA) is fundamental in unsupervised learning.
It generalizes linear ICA (Comon, 1994) to identify latent sources from observations, which are assumed to be a nonlinear mixture of the sources.
For an observed vector x , nonlinear ICA expresses it as x = f ( s ) , where f is an unknown invertible mixing function and s is a latent random vector representing the (marginally) independent sources.
The goal is to recover function f as well as sources s from the observed mixture x up to certain indeterminacies.
While nonlinear ICA is of general interest in a variety of tasks, such as disentanglement (Lachapelle et al., 2022) and unsupervised learning (Oja, 2002), its identifiability has been a long-standing problem for decades.
The key difficulty is that, without additional assumptions, there exist infinite ways to transform the observations into independent components while still mixed w.r.t.
the sources (Hyvärinen and Pajunen, 1999).
To deal with this challenge, existing works introduce the auxiliary variable u (e.g., class label, domain index, time index) and assume that sources are conditionally independent given u (Hyvärinen and Morioka, 2016, 2017; Hyvärinen et al., 2019;
Khemakhem et al., 2020; Lachapelle et al., 2022).
Most of them require the auxiliary variable to be observable, while clustering-based methods (Willetts and Paige, 2021) and those for time series (Hälvä and Hyvärinen, 2020; Hälvä et al., 2021;
Yao et al., 2021) are exceptions.
These works impose mild restrictions on the mixing process but require many distinct values of u for identifiability, which might restrict their practical applicability.
Motivated by this, Yang et al.
(2022) reduce the number of required distinct values by strengthening the functional assumptions on the mixing process.
As described, all these results are based on conditional independence of the sources given auxiliary variable u , instead of the standard marginal 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
independence assumption, to achieve identifiability in specific tasks, thanks to the weak supervision provided by the auxiliary variables.
The identifiability of the original setting without auxiliary variables, which is crucial for unsupervised learning, stays unsolved.
Instead of introducing auxiliary variables, there exists the possibility to rely on further restrictions on the mixing functions to achieve identifiability, but limited results are available in the literature over the past two decades.
With the assumption of conformal transformation, Hyvärinen and Pajunen (1999) proved identifiability of the nonlinear ICA solution up to rotation indeterminacy for only two sources.
Another result is shown for the mixture that consists of component-wise nonlinearities added to a linear mixture (Taleb and Jutten, 1999).
In this work, we aim to show the identifiability of nonlinear ICA with unconditional priors.
We prove that without any kind of auxiliary variables, the latent sources can be identified from nonlinear mixtures up to a component-wise invertible transformation and a permutation under assumptions only on the mixing process.
Specifically, we mainly focus on the structural sparsity condition w.r.t.
the structure from sources to observed variables, i.e., the support of the Jacobian matrix of the mixing function.
Under structural conditions on the mixing process, one could identify the true sources by sparsity regularization during estimation (Thm. 1).
By removing rotation indeterminacy, structural sparsity benefits the identifiability of linear ICA with Gaussian sources as well, which was previously thought to be unsolvable (Prop. 1, Thm. 2).
Besides, we also develop another identifiability theory based on Independent Influences , which is an analogous concept of “sparsity” regarding the individual influencing process instead of the support (Prop. 2).
We establish, to the best of our knowledge, one of the first identifiability results of the original nonlinear ICA model in a fully unsupervised setting.
Moreover, we also provide identifiability results for the undercomplete case, where the number of observed variables is larger than that of sources (Thm. 2, Thm. 3).
This is of great practical interest and cannot be handled with previous conditions on nonlinear ICA, even with the help of auxiliary variables (Hyvärinen et al., 2019).
We validate our theoretical claims experimentally, and the results on the image dataset suggest that our conditions appear to be reasonable for practical data generating processes.
Preliminaries
We consider the following data-generating process of ICA
[Equation]
where n denotes the number of latent sources, x = ( x 1 , .
. . , x n ) denotes the observed random vector, s =
( s 1 , . . .
, s n ) is the latent random vector representing the marginally independent sources, p s i is the marginal probability density function (PDF) of the i -th source
s i , p s is the joint PDF of random vector s , and f : s → x denotes a nonlinear mixing function.
For linear ICA, Eq.
(2) is restricted as
[Equation]
where A :=
[ A : , 1 · · · A : ,n ] denotes the mixing matrix.
The goal of ICA is to learn an estimated unmixing function ˆ f − 1 : x → ˆ s (or alternatively in the linear case, an estimated unmixing matrix ˆ A − 1 ), such that ˆ s =
(ˆ s 1 , . . .
, ˆ s n ) consists of independent estimated sources.
Notations.
Throughout this work, for any matrix M , we use M i, : to refer to its i th row, and M : ,j to indicate its j th column.
For any set of indices S ⊂ { 1 , . . .
, m } × { 1 , . . . , n } , we analogously denote
S i, : := { j | ( i, j ) ∈ S} and S : ,j :=
{ i | ( i, j ) ∈ S} .
We denote by | · | the cardinality of a set.
We also define the following technical notations.
Definition 1.
Given a subset S ⊆ { 1 , . . . , n } , the subspace R n S is defined as
[Equation]
In other words, R n S refers to the subspace of R n indicated by an index set S .
In the following, we define the support of a matrix.
Definition 2.
The support of matrix M ∈ R m × n is defined as
[Equation]
With slight abuse of notation, we also reuse the notation supp to denote the support of a matrix-valued function, depending on the context.
Definition 3.
The support of function M : Θ
→ R m × n is defined as
[Equation]
Identifiability with Structural Sparsity 3.1 Nonlinear ICA with Unconditional Priors: Structural Sparsity
In this section, we consider the identifiability of nonlinear ICA with unconditional priors, where the mixing function f in Eq.
(2) is nonlinear.
Different from recent breakthroughs in identifiable nonlinear
ICA with side information (Hyvärinen et al., 2019; Lachapelle et al., 2022), in our setting, sources s in Eq.
(1) do not need to be mutually conditionally independent given an auxiliary variable.
Instead, our setting is consistent with the original ICA problem based on a general marginal independence assumption of sources.
Besides, no particular form of the source distribution is assumed, which is different from most works that directly assume an exponential family (Hyvärinen et al., 2019).
The goal of identifiable nonlinear ICA is to estimate the unmixing function ˆ f − 1 so that the independent sources are identified up to a certain indeterminacy, which is usually a composition of a componentwise invertible transformation and a permutation (Hyvärinen and Pajunen, 1999).
We formulate the structural sparsity condition below and present the identifiability result with its proof shown in Appx.
A.1.
It is worth noting that Lachapelle et al.
(2022) leverage sparse mechanism between latents and auxiliary variables for disentanglement, by which part of the assumptions and proof technique are inspired.
For brevity, we denote F and ˆ F as the support of the Jacobian J f ( s ) and J ˆ f
(ˆ s ) , respectively.
Theorem 1.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) and (2).
Denote T as the logical matrix corresponding to the support of J f ( s ) − 1 J ˆ f
(ˆ s ) .
Suppose the following assumptions hold:
i. Mixing function f is invertible and smooth.
Its inverse is also smooth. ii.
For all i ∈ { 1 , . . . , n } and j ∈ F i, : , there exist { s ( ℓ ) } |F
i, : | ℓ =1 s.t. span { J f ( s ( ℓ ) )
i, : } |F i, : | ℓ =1 = R n F i, : and (cid:2)
J f ( s ( ℓ ) )
T (cid:3) j, : ∈ R n ˆ F i, : .
iii.
| ˆ F| ≤ |F| .
iv.
(Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Assumption ii in a generic sense rules out a set of specific parameters to avoid ill-posed conditions such as the Jacobian is partially constant and is almost always to be satisfied asymptotically.
Assumption iii corresponds to incorporating a suitable sparsity regularization term into the estimating process with no restriction on the ground truth.
It helps to find the estimated mixing function with the minimal L 0 norm among all functions that allow the model to perfectly fit the true distribution of sources.
Thus, it indicates that we should estimate the mixing process by maximizing its sparsity.
Our discussion will focus on the assumption of structural sparsity, which is the core of our theory.
We start with the intuition of the proposed direction on exploiting sparsity on the mixing process for identifiability.
As discussed in previous works (Hyvärinen and Pajunen, 1999), there exist infinitely many ways to preserve the independence among variables after mixing them up, such as Darmois construction (Darmois, 1951) and measure-preserving automorphism (MPA) (Hyvärinen and Pajunen, 1999).
Besides, sources with non-Gaussian priors could be transferred to marginally independent
Gaussians by trivial point-wise transformations (Hyvärinen et al., 2019).
Thus, the existing strategy of exploiting independence and non-Gaussianity for identification fails in the nonlinear case.
However, if the true structure is sparse enough, any alternative solution that produces any indeterminacy beyond component-wise transformations and permutations (e.g., mixtures or rotations) may correspond to a denser structure.
In the light of that, instead of restricting the functional class (e.g., post nonlinear models (Taleb and Jutten, 1999) or conformal maps (Hyvärinen and Pajunen, 1999)) or introducing auxiliary variables for extra structural dependencies between latent sources (Hyvärinen et al., 2019;
Lachapelle et al., 2022), we leverage the sparsity pattern, i.e., the support of the Jacobian of the mixing function, to identify the sources.
The structural sparsity assumption (Assumption iv in Thm. 1) implies that, for every latent source s i , there exists a set of observed variable(s) such that s i is the only latent source that participates in the generation of all observed variable(s) in the set.
Graphically, for every latent source s i , there exists a set of observed variable(s) such that the intersection of their/its parent(s) is s i (e.g., for s 1 in Fig.
1, there exist x 1 and x 4 such that the intersection of their parents is s 1 .).
This encourages the connections to be sparse enough to make the disentanglement of each source based on structural conditions possible.
Together with the sparsity regularization during estimation (Assumption iii ) and other mild conditions, Assumption iv illustrates a graphical pattern of nonlinear ICA models that could be identified by exploiting structural sparsity.
It indicates a structural criterion of the sparsity that is needed for identifiability.
Beyond that, structural sparsity is also of practical interest for the interpretability of results (Zhang et al., 2009) and identification of other latent variable models (Rhodes and
Lee, 2021).
Meanwhile, various versions of Occam’s razor (e.g., faithfulness (Spirtes et al., 2000), minimality principle (Zhang, 2013), and frugality (Forster et al., 2020)) are also fundamental to the identifiability of the underlying causal structure.
Readers may refer to additional discussion in Sec. 6.
With structural sparsity, Thm. 1 shows the identifiability of nonlinear ICA while maintaining the standard mutually marginal independence assumption of sources.
This is consistent with the original setting of ICA and plays an important role in a more general range of unsupervised learning tasks, compared to previous works relying on conditional independence given auxiliary variables.
Moreover, one may modify Thm.
1 to consider the generating process based on auxiliary variables by imposing similar restrictions on the mixing process.
Different from previous works (Hyvärinen et al., 2019;
Yang et al., 2022), this extension removes common restrictions on the required auxiliary variables, such as the number of distinct values (e.g., class labels), non-proportional variances, or sufficient variability.
Therefore, although our main result (Thm. 1) focuses on nonlinear ICA with unconditional priors, it also provides potential insight into improving the flexibility of utilizing the auxiliary variable when it is available.
As a trade-off, the additional assumptions on the sparsity might limit its usage.
Besides, under additional assumptions, we could reduce the indeterminacy in Thm. 1 and further identify sources up to a component-wise linear transformation.
Most works in nonlinear ICA focus on the identifiability up to a component-wise invertible transformation and a permutation (Hyvärinen and Morioka, 2016; Khemakhem et al., 2020), which is also called permutation-identifiability in the literature of disentanglement (Lachapelle et al., 2022).
This indeterminacy is trivial compared to the fundamental nonuniqueness of nonlinear ICA and is analogous to the indeterminacy involving rescaling and permutation in linear ICA (Hyvärinen and Pajunen, 1999).
Recently, Yang et al.
(2022) provided an identifiability result based on auxiliary variables that further reduces the indeterminacy of the component-wise nonlinear transformation to a linear one.
Inspired by it but without auxiliary variables, we show conditions for the identifiability of nonlinear ICA with unconditional priors up to a component-wise linear transformation, with the proof in Appx.
A.2.
The reduced indeterminacy may give rise to a more informative disentanglement and open up the possibility for further improving the quality of recovery with only element-wise linear operations, such as the Hadamard product.
Corollary 1.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) and (2) .
Suppose the following assumptions hold:
i.
The function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
ii.
The mixing function f is volume-preserving.
iii.
The source distribution p s ( s ) is a factorial multivariate Gaussian.
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise linear transformation and a permutation.
3.2 Removing Rotation Indeterminacy with Structural Sparsity
Rotation indeterminacy is one of the major obstacles to the identifiability of ICA.
Linear ICA exploits the maximization of the non-Gaussianity of the estimated sources (e.g., Kurtois (Hyvärinen and Oja, 1997)).
As a direct result, the typical assumption is that at most one of the sources can have Gaussian distribution.
In contrast, for the nonlinear case, a trivial point-wise function could transform sources to have any marginal distribution including Gaussian thus invalidating the effect of non-Gaussianity.
Therefore, removing rotation indeterminacy remains critical for the identifiability in the nonlinear case.
For instance, “rotated-Gaussian” MPA produces nonuniqueness in nonlinear ICA due to rotation indeterminacy (Hyvärinen and Pajunen, 1999).
It first maps the distribution to an isotropic Gaussian, then applies a rotation and maps it back without affecting its original distribution.
To address it for identifiability, Yang et al.
(2022) assume that there do not exist two conditionally independent
Gaussians such that their variances are proportional.
In the previous section, we have introduced the structural sparsity for the identifiability of nonlinear
ICA.
One may wonder whether it straightforwardly leads to the identifiability of linear ICA with
Gaussian sources (Gaussian ICA), which was previously perceived to be impossible due to its rotation indeterminacy.
The idea is simple–any rotation of the true mixing matrix A will be less sparse if A satisfies the proposed structural sparsity assumption.
While the joint distribution of Gaussian sources stays invariant across rotations, the sparsity of the mixing matrix (i.e., L 0 norm) keeps changing.
We first consider Assumption iv in Thm. 1 for Gaussian ICA as an extension.
Because the Jacobian of linear mixing function is fixed, Assumption ii of Thm.
1 does not directly hold in the linear Gaussian case.
As a result, we cannot directly apply Thm.
1 here
and therefore propose some alternative conditions:
Proposition 1.
Let the observed data be sampled from a linear ICA model defined in Eqs.
(1) and (3) with Gaussian sources.
Suppose the following assumptions hold:
i.
Mixing matrix A is invertible.
ii.
There exists a matrix ˆ A s.t.
for all j ∈ supp( A ) i, : , supp( ˆ AA − 1 ) j, : ∈ R n supp(ˆ A ) i, : . iii.
| supp( ˆ A ) | ≤ | supp( A ) | .
iv.
(Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that (cid:92) i ∈C k supp
( A i, : )
= { k } .
Then ˆ A = ADP , where D is a diagonal matrix and P is a column permutation matrix.
The proof is provided in Appx.
A.3.
Similar to Assumption ii of Thm. 1, Assumption ii of Prop.excludes a specific set of parameters that makes structure-based identification ill-posed.
The undercomplete case.
For the full identifiability of nonlinear ICA, it is noteworthy that the invertibility of the mixing function is technically essential in our result (Thm. 1).
This invertibility implies that the number of observed variables is equal to that of sources.
Although that setting is rather common in the literature, especially for the theory of nonlinear cases (Hyvärinen and Pajunen, 1999; Hyvärinen et al., 2019), undercomplete ICA (i.e., there are more observed variables), may be of more practical interest and more challenging (Joho et al., 2000).
Therefore, we develop an alternative structural sparsity condition for the undercomplete case.
As a trade-off, in the nonlinear case, the structural condition that removes the requirement of invertibility could only deal with the rotation indeterminacy.
Thus we start from Gaussian ICA to introduce the condition.
Definition 4.
Given a matrix S ∈ R m × n , we define the function overlap : R m × n → { 0 , 1 } m × n as (overlap( S )) ij = (cid:26)if S ij =[-
-]
1 and it is not the only nonzero entry in S i, : otherwise.
Theorem 2.
Let the observed data be sampled from a linear ICA model defined in Eqs.
(1) and (3) with Gaussian sources.
Differently, the number of observed variables (denoted as m ) could be larger than that of the sources n , i.e., m ≥ n .
Suppose the following assumptions hold:
i.
The nonzero coefficients of the mixing matrix A are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure.
ii.
The estimated mixing matrix ˆ A has the minimal L 0 norm during estimation.
iii.
(Structural Sparsity)
Given C ⊆ { 1 , 2 , . . .
, n } where |C| > 1 , let A C ∈ R m ×|C| represents a submatrix of A ∈ R m × n consisting of columns with indices C .
Then, for all k ∈ C , we have
[Equation]
Then ˆ A =
ADP with probability one, where D is a diagonal matrix and P is a column permutation matrix.
The proof is provided in Appx.
A.4.
Intuitively, Assumption iii in Thm.encourages the variability between the influences of each source.
For each source, its influence on the observations should be as distinctive as possible, which is correlated with the sparsity of the mixing matrix.
Fig.
2 is an illustration of the assumption.
In that case, k = 1 and C = { 1 , 2 , 3 } .
Let A C denotes the submatrix shown in the figure, where (cid:12)(cid:12)(cid:12)(cid:12) (cid:83)
[Equation]
Blue dotted square denotes overlap( A C ) and we have rank(overlap( A C ))
= 2 .
Thus, (cid:12)(cid:12)(cid:12)(cid:12) (cid:83) k ′ ∈C supp( A k ′ ) (cid:12)(cid:12)(cid:12)(cid:12) − rank(overlap( A C ))
=
5 (the number of black dots), which is larger than | supp( A 1 ) | = 4 .
For the nonlinear case, we prove that it could help to distinguish spurious solutions due to the rotation indeterminacy, i.e., the “rotated-Gaussian” MPA (Defn.
2.5 in Gresele et al.
Gresele et al.
(2021) prove that this class of MPAs could be ruled out with assumptions of conformal maps, non-Gaussianity and orthogonality of the Jacobian of the mixing function.
Differently, we address it with the proposed structural sparsity condition (Assumption iii in Thm. 2).
The corresponding theorem for the nonlinear case, with its proof given in Appx.
A.5, is as follows:
Theorem 3.
Given a nonlinear ICA model defined in Eqs.
(1) and (2) , where f is the true mixing function.
Consider ˆ f = f ◦ G − 1 ◦
U ◦ G , where G denotes an invertible Gaussianization 1 that maps the distribution to an standard isotropic (rotation-invariant) Gaussian, U denotes a rotation, and G − 1 maps the distribution back to that before applying ˆ U ◦ G .
If Assumptions i, ii and iii of
Thm. 2 are satisfied by replacing A with J f ( s ) and ˆ A with J ˆ f ( s ) , then function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation with probability one.
Besides removing rotation indeterminacy, Thm. 3 also provides an extra insight for the full identifiability of nonlinear ICA, because all previous works assume that the nonlinear mixing function must be invertible (Hyvärinen and Morioka, 2016; Yang et al., 2022) even with the help of auxiliary variables.
Identifiability with Independent Influences
In the previous section, we have presented theoretical results based on conditions of structural sparsity.
Alternatively, we show in this section that sparsity conditions on the support might not be necessary in some specific scenarios, and that one can leverage the independence (in a non-statistical sense) among the influences from different sources to the observations to achieve identifiability of nonlinear ICA.
This implies sparse interactions between the individual influencing processes and could be viewed as an analogous concept of “sparsity” w.r.t.
influences.
We show that without any kind of auxiliary variables, the full identifiability of nonlinear ICA can also be provided given the condition of independent influences, thus complementing the results shown above from a different view of “sparsity”.
Nonlinear ICA with unconditional priors: Independent Influences.
We now specify the setting of the required estimating process.
We sample the estimated sources ˆ s from a multivariate Gaussian:
[Equation]
where Z i > 0 is a constant.
The sufficient statistics θ i, 1 = − µ i σ
i
and θ i, 2 = 12 σ 2 i are assumed to be linearly independent.
We constraint the variances σ 2 i to be distinct without loss of generality.
Proposition 2.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) , (2) , and (7) .
Suppose the following assumptions hold
: i. (Independent Influences): The influence of each source on the observed variables is independent of each other, i
e., each partial derivative ∂ f /∂s
i is independent of the other sources and their influences in a non-statistical sense.
ii.
The mixing function f and its inverse are twice differentiable.
iii.
The Jacobian determinant of mixing function can be factorized as det( J f ( s ))
=
(cid:81) ni
=
y i
( s i ) , where y i is a function that depends only on s i .
Note that volume-preserving transformation is a special case when y i ( s i )
= 1 , i = 1 , . .
. , n .
iv.
During estimation, the columns of the Jacobian of the estimated unmixing function are regularized to be mutually orthogonal and with
equal euclidean norm.
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
The proof is included in Appx.
A.6.
Note that some part of the proof technique is inspired by or based on Lemma 1 in Yang et al.
Assumption iv is a regularization during estimation and thus puts no restriction on the true generating process.
We achieve it by optimizing an objective function with a designed regularization term, of which the property is as follows (proof in Appx.
A.7):
Proposition 3.
The following inequality holds
[Equation]
[Table]
[Equation]
[Table]
with equality iff.
J ˆ f − 1 ( x ) =
O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar.
Regarding the assumption of independent influences, to build intuition, one can consider a filmmaking process in a safari park, of which the main characters are wild animals.
The task of the boom operators here is to record the sound of animals.
As there are many animals and microphones, the mixing (recording) process is highly dependent on the relative positions between them.
Thus, the animal (source s i ), loosely speaking, influences the recording (mixing function f ) through its position.
The moving direction and speed of them can be loosely interpreted as the partial derivative w.r.t.
the i -th source, i.e. ∂ f /∂s i .
Assuming that the wild animals are not cooperative enough to fine-tune their positions and speeds for a better recording and the safari park is not crowded, the influences of animals on the recording are generated independently by them in the sense that they are not affected by the others while moving.
As a result of that generating process, the column vectors of the Jacobian of the mixing function are uncorrelated with each other and the partial derivative w.r.t.
to each source is independent of other sources.
Other practical scenarios include fields that adopt independent influences in process like orthogonal coordinate transformations (Gresele et al., 2021), such as dynamic control (Mistry et al., 2010) and structural geology (De Paor, 1983).
It is worth noting that this assumption of independent influences is different from the orthogonality condition proposed in (Gresele et al., 2021).
For instance, if the sample mean is not zero, uncorrelatedness differs from orthogonality.
And we argue that uncorrelatedness as a specific way to achieve independence might be more sensible as a condition on the influences.
Besides, in (Gresele et al., 2021), orthogonality, together with conformal map and the others, are proposed as conditions to rule out two specific types of spurious solutions (i.e., Darmois construction (Darmois, 1951) and “rotated-Gaussian” MPAs (Locatello et al., 2019)) without an identifiability result, although its empirical results are encouraging.
Having said that, a violation of the assumption of independent influences could be ascribed to a deliberate global adjustment of sources.
For example, cinema audio systems are carefully adjusted to achieve a homogeneous sound effect on every audience.
The position and orientation of each speaker are fine-tuned according to the others.
In this case, it may be difficult to distinguish the influences from different speakers because of the fine-tuning.
This may lead to a high degree of multicollinearity across the columns of Jacobian, thus violating the assumption of independent influences.
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(a) SS .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(b) II .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(c) VP .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(d) Base .
Regarding Assumption iii in Prop. 2, we first note that mixing functions with factorial Jacobian determinants are of a much wider range as compared to component-wise transformations.
To see this, a straightforward example is the volume-preserving transformation, which has been widely adopted in generative models (Zhang et al., 2021).
In fact, all transformations with constant Jacobian determinant can be factorized w.r.t.
the sources, which, generally speaking, are not component-wise.
Besides, consistent with the empirical results in Yang et al.
(2022), there exists other non-volume-preserving transformations with factorial Jacobian determinant 2 .
Moreover, the volume-preserving assumption has been demonstrated to be helpful to weaken the requirement of auxiliary variables, specifically by reducing the number of required labels (Sorrenson et al., 2020; Yang et al., 2022).
Prop. 2 takes this further by imposing a weaker version, i.e., Assumption (iii), which, together with the other constraints, removes the need for any auxiliary variable.
Experiments
To validate the proposed theory of the identifiability of nonlinear ICA with unconditional priors, we conduct experiments based on our main condition, i.e., structural sparsity (Thm. 1), as well as the condition of the independent influences (Prop. 2).
Setup.
For the required regularization during estimation, we consider a regularized maximumlikelihood approach with the following objective:
[Equation]
[Equation]
, where λ is a regularization parameter and R (ˆ f − 1 , x ) is the regularization term.
For Thm. 1, we use
L 1 norm as an approximation of the L 0 norm for efficiency (Donoho and Elad, 2003), therefore
R (ˆ f − 1 , x ) := ∥
J ˆ f − 1 ( x ) ∥ 1 ; for Prop. 2, R (ˆ f − 1 , x ) corresponds to LHS of Eq.
Following (Sorrenson et al., 2020; Yang et al., 2022), we train a GIN (Sorrenson et al., 2020) to maximize the the objective function L (ˆ f − 1 ; x ) .
[Figure]
Ablation study.
We conduct an ablation study to verify the necessity of the proposed assumptions.
Specifically, we focus on the following models that correspond to different assumptions:
(SS)
The assumption of structural sparsity, as well as other assumptions in Thm. 1, are satisfied; (II) The assumption of independent influences, as well as other assumptions in Prop. 2, are satisfied; (VP)
Compared to II , only the assumption of independent influences is violated while the other assumptions (e.g., factorizable Jacobian determinant) are still satisfied; (Base)
The vanilla baseline.
Compared to VP , the (un)mixing function is not restricted to having factorizable Jacobian determinants
The data are generated according to the required assumptions.
We also conduct comparison between the assumption of independent influences and orthogonality in (Gresele et al., 2021), which are presented in Appx.
B together with experimental settings.
Results for each model are summarised in Fig.
For evaluation, we use the mean correlation coefficient (MCC) between the true sources and the estimated ones (Hyvärinen and Morioka, 2016).
One could observe that when the proposed assumptions are fully satisfied ( SS or II ), our model achieves the highest
MCC on average.
This indicates that it is actually possible to identify sources from highly nonlinear mixtures up to trivial indeterminacies only based on restrictions on the mixing process.
Moreover, regarding Prop. 2, the importance of Assumption iii is supported by the higher performance of VP compared to that of Base .
The visualization of the Pearson correlation matrices is shown in Fig.
[Figure]
Stability.
To study the stability of the performance of identification w.r.t.
different datasets varying the number of sources n .
We test the model SS (Thm. 1) with different n .
Visually, we find that SS consistently outperforms
Base (Fig. 5).
Meanwhile, when the number of sources increases, one could observe that the
MCC of Base decreases while that of SS stays stable.
The stable empirical performance further validates our theoretical claims about the identifiability with structural sparsity.
To study how reasonable the proposed theories are w.r.t.
the practical generating process of observational data
, we conduct experiments on the “Triangles" image dataset (Yang et al., 2022).
The process of generating this dataset mimics the process of drawing triangles by humans: i)
First, we sample the elements needed for humans to draw a monochrome triangle (i.e., rotation, width, height and grey level) from a factorial multivariate Gaussian distribution.
Different from (Yang et al., 2022), we always sample from a single distribution in order to guarantee that all priors are unconditionally independent; ii) Then, for each pixel, we decide whether it locates inside the triangle based on the sampled elements and assign its gray level accordingly.
Therefore, the process is similar to human drawing triangles.
Even though each image is generated from these semantic elements, the true generating process and sources are still unknown (e.g., a pixel could be (indirectly) influenced by multiple elements in a complicated way).
We apply GIN with sparsity regularization as the estimating method.
The visualization of the identified sources (Fig. 6) indicates that our conditions may hold in practice.
[Figure]
IMA.
Recently, Gresele et al.
(2021) assume orthogonality, conformal map, and the others in order to rule out Darmois construction and “rotated-Gaussian” MPAs.
They formalize the orthogonality between columns of the Jacobian of the mixing function as independent mechanism analysis ( IMA ) and show empirically that it improves the identification of latent sources.
To further explore their exciting results, we generate datasets according to the generating process and regularization term described in (Gresele et al., 2021).
We sample the sources from the same distributions described in Appx.
B and use GIN for training.
From Fig.
7, one could observe that indeed IMA outperforms the baseline ( Base ) largely, which indicates that this condition is empirically helpful to the identifiability of nonlinear
ICA.
At the same time, MCC of II appears to be even higher than that of IMA , which might thanks to some additional constraints, such as the factorial volume.
Meanwhile, it also supports the conjecture that IMA is a sensible condition for identification, and the identifiability based on it may be achieved with some additional assumptions, such as conformal maps (Gresele et al., 2021).
Discussion and Conclusion
Sparsity Assumptions.
Sparsity assumptions have been widely used in various fields.
For latent variable models, sparsity in the generating process plays an important role in the disentanglement or identification of latent factors both empirically and theoretically (Bing et al., 2020;
Rohe and Zeng, 2020;
Moran et al., 2021; Rhodes and Lee, 2021; Lachapelle et al., 2022).
In causality, various versions of
Occam’s razor have been proposed to serve as fundamental assumptions for identifying the underlying causal structure (Spirtes et al., 2000; Zhang, 2013; Raskutti and Uhler, 2018; Forster et al., 2020).
Formulated as a measure of the density of dependencies, sparsity assumptions are more likely to be held when the observations are actually influenced by the sources in a “simple” way.
For example, in biology, analyses on ecological, gene-regulatory, metabolic, and other living systems find that active interactions may often be rather sparse (Busiello et al., 2017), even when these systems evolve with an unlimited number of complicated external stimuli.
In physics, it is an important heuristic that a relatively small set of laws govern complicated observed phenomena.
For instance, Einstein’s theory of special relativity contains parsimonious relations between substances as an important heuristic to shave away the influence of ether compared to Lorentz’s theory (Einstein, 1905; Nash, 1963).
However, sparsity is not an irrefutable principle and our assumption may fail in a number of situations.
The most direct one could be a scenario with heavily entangled relations between sources and observations.
Let us consider the example of animal filmmaking in Sec. 4, where people are recording the sound of animals in a safari park.
If the filming location is restricted to a narrow area of the safari park and multiple microphones are gathered together, the recording of each microphone will likely be influenced by almost all animals.
In such a case, the dependencies between the recording of microphones and the animals are rather dense and our sparsity assumption is most likely not valid.
At the same time, even when the principle of simplicity holds, our formulation of sparsity, which is based on the sparse interactions between sources and observations, may still fail.
One reason for this is the disparity between mechanism simplicity and structural sparsity.
To illustrate this, one could consider the effect of sunlight on the shadow angles at the same location.
In this case, the sun’s rays and the shadow angles act as the sources and the observations, respectively.
Because rays of sunlight, loosely speaking, may be parallel to each other, the processes of them influencing the shadow angles may be almost identical.
Thus, the influencing mechanism could be rather simple.
On the other hand, each shadow angle is influenced by an unlimited number of the sun’s rays, which indicates that the interactions between them may not be sparse, therefore violating our assumption.
This sheds light on one of the limitations of our sparsity assumption, because the principle of simplicity could be formulated in several ways.
Besides, these different formulations also suggest various possibilities for identifiability based on simplicity assumptions.
Another proposed assumption, i.e., independent influences, may be one of the alternative formulations, and more works remain to be explored in the future.
Conclusion.
We provide identifiability results for nonlinear ICA with unconditional priors, which serve as one of the first steps to solve a long-standing problem in unsupervised learning.
In particular, we prove that the i.i.d.
latent sources can be recovered up to a component-wise invertible transformation and a permutation with only conditions on the nonlinear mixing process (e.g., structural sparsity).
Therefore it stays closer to the original notion of ICA that is based on the marginal independence assumption of latent sources, while previous works rely on conditional independence on auxiliary variables as weak supervision or inductive bias.
Besides, by removing rotation indeterminacy, structural sparsity benefits the identifiability of Gaussian ICA as well, which was also thought to be unsolvable before.
Moreover, the results on the undercomplete case are of great practical interest and introduce insight for extending identifiable nonlinear ICA to general real-world settings.
Our results on images illustrate the validity of the proposed conditions in practical data generating processes.
In spite of this, it is possible that part of them is violated in several specific scenarios as discussed before.
For example, the structural sparsity conditions do not apply to fully-connected structures, though the practical significance of such cases may be compromised by the lack of interpretability.
As a complementary solution, we formulate the independent influences condition, which does not rely on the sparse structure of supports.
While arguably natural in general, it could still be violated due to a deliberate global adjustment of sources for homogeneity.
We argue that this is inevitably a trade-off between introducing auxiliary variables and imposing restrictions on the mixing process to achieve the identifiability, whose practical use depends on the scenario and information available.
Future work includes further generalizing and validating our theory.
Acknowledgements
We thank the anonymous reviewers for their constructive comments.
This work was partially supported by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSFConvergence Accelerator Track-D award #2134901, by a grant from Apple Inc., and by a grant from
KDDI Research Inc..
A Proofs A.1 Proof of Theorem
Theorem 1.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) and (2).
Denote T as the logical matrix corresponding to the support of J f ( s ) − 1 J ˆ f
(ˆ s ) .
Suppose the following assumptions hold:
i. Mixing function f is invertible and smooth.
Its inverse is also smooth. ii.
For all i ∈ { 1 , . . . , n } and j ∈ F i, : , there exist { s ( ℓ ) } |F
i, : | ℓ =1 s.t. span { J f ( s ( ℓ ) )
i, : } |F i, : | ℓ =1 = R n F i, : and (cid:2)
J f ( s ( ℓ ) )
T (cid:3) j, : ∈ R n ˆ F i, : .
iii.
| ˆ F| ≤ |F| .
iv.
(Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that
[Equation]
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Proof.
Our goal here is to show that function h :
= ˆ f − 1 ◦ f is a permutation with component-wise invertible transformation of sources, i.e., ˆ f = f ◦ h − 1 ( s ) .
Let D ( s ) represents a diagonal matrix and P represent a permutation matrix.
By using chain rule repeatedly, we write ˆ f = f ◦ h − 1 ( s ) equivalently as
[Equation]
where g is an invertible element-wise function.
Thus our goal is equivalent to show that
[Equation]
Because J ˆ f (ˆ s ) and J f ( s ) are both invertible, we have the following equation
[Equation]
where T ( s ) is an invertible matrix.
Note that we denote F as the support of J f ( s ) , ˆ F as the support of J ˆ f
(ˆ s ) and T as the support of
T ( s ) .
Besides, we denote T as the logical matrix with indices of ones corresponding to the support
T .
According to Assumption ii, we have
[Equation]
Since { J f ( s ( ℓ ) )
i, : } |F i, : | ℓ =1 forms a basis of R n F i, : , for any j 0 ∈ F i, : , we are able to rewrite the one-hot vector e j 0 ∈ R n F i, : as
[Equation]
where α ℓ is the corresponding coefficient.
Then
[Equation]
where the final “ ∈ ” follows from Assumption ii that each element in the summation belongs to R n ˆ F
i, : .
Thus
[Equation]
Then we are able to draw connections between these supports according to Defn.
[Equation]
According to Assumption i, T ( s ) is an invertible matrix, indicating that it has a non-zero determinant.
Representing the determinant of the matrix T ( s ) as its Leibniz formula yields
[Equation]
where S n is the set of n -permutations.
Then there exists at least one term of the sum that is non-zero, i.e.,
[Equation]
This is equivalent to
[Equation]
Then we can see that this σ is in the support of T ( s ) , which implies that
[Equation]
Together with Eq.
(16), it follows that
[Equation]
Denote
[Equation]
Then we have
[Equation]
According to Assumption iii, we can see that
[Equation]
Together with Eq.
(23), we have
[Equation] [Equation]
Suppose T ( s ) ̸
= D ( s ) P , then
[Equation]
[Equation]
Besides, consider j 3 ∈ { 1 , . .
. , n }
such that
[Equation]
Because j 1 ̸ =
j 2 , we can assume j 3 ̸ =
j 1 without loss of generality.
A similar strategy has previously been used in (Lachapelle et al., 2022).
By Assumption iv, there exists C j 1 ∋ j 1 such that (cid:84)
i
[Equation]
1 F i, : = { j 1 } .
Because
[Equation]
there must exists i 3 ∈ C j 1 such that
[Equation]
Because j 1 ∈ F i 3 , : , we have ( i 3 , j 1 ) ∈ F .
Then according to Eq.
(16), we have the following equation
[Equation]
Notice that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies
[Equation]
Then by Eqs.
(31) and (30), we have
[Equation]
) which implies ( i 3 , j 3 ) ∈ F by Eq.
(25) and Eq.
(22), therefore contradicting Eq.
Thus, we prove by contradiction that T ( s ) =
D ( s ) P .
Replacing T ( s ) with D ( s ) P in Eq.
(11), we prove Eq.
(10), which is the goal.
A.2 Proof of Corollary
Corollary 1.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) and (2) .
Suppose the following assumptions hold:
i.
The function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
ii.
The mixing function f is volume-preserving.
iii.
The source distribution p s ( s ) is a factorial multivariate Gaussian.
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise linear transformation and a permutation.
Proof.
The proof technique of this corollary is inspired by Thm. 2 in Yang et al.
According to Assumption iii, true sources s are from a factorial multivariate Gaussian distribution.
Together with the estimated sources
ˆ s from the same type of distribution, we represent the densities of the true and estimated sources as
[Equation]
) where Z i > 0 is a constant.
The sufficient statistics θ i, 1 and θ i, 2 are assumed to be linearly independent.
Applying the change of variable rule, we have p s ( s ) =
p ˆ s (ˆ s ) | det( J h ( s ))
| , which, by plugging in Eq.
(33) and taking the logarithm on both sides, yields
[Equation]
Because of Assumption ii and the corresponding estimating process, det( J h ( s ))
= 1 .
Thus
[Equation]
According to Assumption i, function h is a component-wise invertible transformation of sources, i.e., h i ( s )
=
h i ( s i ) .
Combining Eqs.
(33) and (36) with Eq.
(35), it follows that
[Equation]
Then we have
[Equation]
Therefore, h i ( s i ) is a linear function of s i .
A.3 Proof of Proposition
Proposition 1.
Let the observed data be sampled from a linear ICA model defined in Eqs.
(1) and (3) with Gaussian sources.
Suppose the following assumptions hold:
i.
Mixing matrix A is invertible.
ii.
There exists a matrix ˆ A s.t.
for all j ∈ supp( A ) i, : , supp( ˆ AA − 1 ) j, : ∈ R n supp(ˆ A ) i, : . iii.
| supp( ˆ A ) | ≤ | supp( A ) | .
iv.
(Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that
[Equation]
Then ˆ A = ADP , where D is a diagonal matrix and P is a column permutation matrix.
Proof.
Let ˆ A
=
AT , where T ∈ R n × n is an arbitrary matrix.
Because our goal is to prove ˆ A
= ADP , it is equivalent to prove the following equation:
[Equation]
For brevity of notation, we denote A as the support of ˆ A , ˆ A as the support of A and T as the support of T .
According to Assumption ii, we have
[Equation] [Equation]
which is equivalent to
[Equation]
Then we have
[Equation]
According to Assumption i, T is an invertible matrix, indicating that it has a non-zero determinant.
Representing the determinant of the matrix T as its Leibniz formula yields
[Equation]
where S n is the set of n -permutations.
Then there exists at least one term of the sum that is non-zero, i.e.,
[Equation]
This is equivalent to
[Equation]
Then we can see that this σ is in the support of T , which implies that
[Equation]
Together with Eq.
(40), it follows that
[Equation]
Denote
[Equation]
Then we have
[Equation]
According to Assumption iii, we can see that
[Equation]
Together with Eq.
(47), we have
[Equation]
Suppose T ̸ = DP , there must exists j 1 and j 2 such that
[Equation]
Besides, consider j 3 ∈ { 1 , . .
. , n }
such that
[Equation]
Because j 1 ̸ =
j 2 , we can assume j 3 ̸ =
j 1 without loss of generality.
By Assumption iv, there exists
C j 1 ∋ j 1 such that (cid:84)
i ∈C j 1 A i, : = { j 1 } .
Because
[Equation]
there must exists i 3 ∈ C j 1 such that
[Equation]
Because j 1 ∈ A i 3 , : , we have ( i 3 , j 1 ) ∈ A .
Then according to Eq.
(40), we have the following equation
[Equation]
Notice that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies
[Equation]
Then by Eqs.
(55) and (54), we have
[Equation]
which implies ( i 3 , j 3 )
∈
A by Eqs.
(49) and (46), contradicting Eq.
Thus, we prove by contradiction
that T = DP , which is the goal (i.e., Eq.
A.4 Proof of Theorem
Theorem 2.
Let the observed data be sampled from a linear ICA model defined in Eqs.
(1) and (3) with Gaussian sources.
Differently, the number of observed variables (denoted as m ) could be larger than that of the sources n , i.e., m ≥ n .
Suppose the following assumptions hold:
i.
The nonzero coefficients of the mixing matrix A are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure.
ii.
The estimated mixing matrix ˆ A has the minimal L 0 norm during estimation.
iii.
(Structural Sparsity)
Given C ⊆ { 1 , 2 , . . .
, n } where |C| > 1 , let A C ∈ R m ×|C| represents a submatrix of A ∈ R m × n consisting of
columns with indices C .
Then, for all k ∈ C , we have
[Equation]
Then ˆ A =
ADP with probability one, where D is a diagonal matrix and P is a column permutation matrix.
Proof.
Because of Assumptions ( ii), we consider the following combinatorial optimization
[Equation]
where A is the true mixing matrix and ˆ U denotes the rotation matrix corresponding to the solution of the optimization problem.
Let ˆ A
=
A ˆ U .
Suppose ˆ A ̸
=
ADP , then ˆ U ̸ = DP .
This implies that there exists some j ′ ∈ { 1 , . . .
, s } and its corresponding set of row indices
I j ′ ( |I j ′ | > 1 ), such that ˆ U
i,j ′ ̸ = 0 for all i ∈ I ′ j , and ˆ U i,j ′ = 0 for all i / ∈ I j ′ .
Because ˆ U is invertible and has full row rank, there exists one row index i ′ in I j ′ that uniquely correspond to j ′ , in order to avoid linear dependence among columns.
Let ˆ U :=
(cid:104)
ˆ U 1 · · · ˆ U s (cid:105) , we have (cid:13)(cid:13)(cid:13)
[Equation]
Let A I
j ′ ∈
R m × | I j ′ | represents a submatrix of A consisting of columns with indices I j ′ .
Note that with a slight abuse of notation, A i denotes i -th column of the matrix A .
According to Assumptions (i, iii), since |I j ′ | > 1 and ˆ U i,j ′ ̸
= 0 , we have
[Equation]
59) where overlap( · ) is defined as Defn. 4.
Term rank(overlap( A I j ′ )) represents the maximal number of rows, in which all non-zero entries can be possibly cancelled out by the linear combination (cid:80) i ∈I
j ′ A i .
Assumption i rules out a specific set of parameters that leads to a violation of that, e.g., two columns of A are identical in terms of element values and support.
So it follows that
[Equation]
[Table]
[Equation]
Then we can construct ˜ U := (cid:104) ˜
U 1 · · · ˜ U s (cid:105) .
First, we set ˜ U
i ′ ,j ′ as a unique non-zero entry in column ˜
U j ′ .
For simplicity, we can just set ˜ U
i ′ ,j ′
= 1 .
For other column ˜
U j , where j ̸ = j ′ and ˆ U i,j ̸ = 0 , we set ˜
U
i,j = 1 .
Therefore
[Equation]
Since Assumption iii covers all columns, Eq.
(60) holds for any j ′ ∈ { 1 , . . .
, s } .
If there are multiple columns of ˆ U with more than one nonzero entry, we derive Eq.
(60) for each of them.
We denote the set of different target column indices j ′ as J .
For j ∈ J , we set ˜
U
i j ,j = 1 , where i j is the unique index of the corresponding non-zero entry in column ˆ U j .
For other column ˜
U j , where j / ∈ J and ˆ U
i,j ̸ = 0 , we set ˜
U
i,j = 1 .
Then we have
[Equation]
As noted previously, every column index j corresponds to a unique row index.
˜ U is a permutation matrix and ˜ U ˜
U
⊤ =
I s .
It then follows that (cid:13)(cid:13)(cid:13)
A ˆ U (cid:13)(cid:13)(cid:13) 0 > (cid:13)(cid:13)(cid:13)
A ˜ U (cid:13)(cid:13)(cid:13) 0 , which contradicts the definition of ˆ U .
A.5 Proof of Theorem
Theorem 3.
Given a nonlinear ICA model defined in Eqs.
(1) and (2) , where f is the true mixing function.
Consider ˆ f = f ◦ G − 1 ◦
U ◦ G , where G denotes an invertible Gaussianization 3 that maps the distribution to an standard isotropic (rotation-invariant) Gaussian, U denotes a rotation, and G − 1 maps the distribution back to that before applying ˆ U ◦ G .
If Assumptions i, ii and iii of
Thm. 2 are satisfied by replacing A with J f ( s ) and ˆ A with J ˆ f ( s ) , then function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation with probability one.
Proof.
Let D ( s ) represents a diagonal matrix and P represent a permutation matrix.
By using the chain rule repeatedly, we write ˆ f = f ◦ h ( s ) equivalently as
[Equation]
where g is an invertible element-wise function.
Thus our goal is equivalent to show that
[Equation]
We prove it by contrapositive.
Because ˆ f = f ◦ G − 1 ◦ U ◦ G , we write
[Equation]
where we have used the chain rule repeatedly.
Because G is an invertible element-wise transformation,
D 1 ( s ) and D 2 ( s ) are both diagonal matrices.
Because Assumption ii of Thm.
2 is satisfied for J f ( s ) , we consider the following combinatorial optimization problem
[Equation] [Equation]
Let ¯ U
= D 1 ( s )
ˆ UD 2 ( s ) .
Because J ˆ f ( s ) =
J f ( s ) ¯ U , then if J (ˆ f ) =
J f ( s ) D ( s ) P , we have ¯ U =
D ( s ) P .
Thus, for every j ∈ { 1 , . .
. , s } , there exists a corresponding i ′ , such that ¯ U i ′ ,j ̸ = 0 , and ¯
U i,j = 0 for all i ̸
= i ′ .
Because the columns of the matrix of an orthogonal transformation form an orthogonal set, columns of ¯ U are linearly independent.
Thus, different j cannot correspond to the same i ′ , otherwise it is possible for these columns to be linearly dependent.
Suppose J ˆ f ( s ) ̸
= J f ( s ) P , then ¯
U ̸
= P .
There exists j ′ ∈ { 1 , . . .
, s } and its corresponding set of row indices
I j ′ ( |I j ′ | > 1 ),
such that ¯ U i,j ′ ̸ = 0 for all i ∈ I ′ j , and ¯ U i,j ′ =
for all i / ∈ I j ′ .
Similarly, there exists one row index i ′ in I j ′ that uniquely correspond to j ′ , in order to avoid linear dependence among columns.
Let ¯ U := (cid:2) ¯ U 1 · · · ¯ U s (cid:3) , we have
[Equation]
[Equation]
Let J f ( s ) I j ′ ∈
R m × | I j ′ | represents a submatrix of J f ( s ) consisting of columns with indices I j ′ .
Note that with a slight abuse of notation, J f ( s ) i denotes i -th column of the matrix J f ( s ) .
Because
Assumptions (i, iii) of Thm.
2 hold for J f ( s ) , with |I j ′ | >
and ¯ U i,j ′ ̸
= 0 , we have
[Equation]
where overlap( · ) is defined as Defn.
Note that here we slightly abuse the notation overlap( · ) to make it apply for the matrix-valued function J f ( s ) .
Term rank(overlap( J f ( s ) I j ′ )) generally represents the maximal number of rows, in which all non-zero entries can be cancelled out by the linear combination (cid:80) i ∈I
j ′ J f ( s ) i .
Also with a slight abuse of the notation, Assumption i of Thm.rules out a specific set of parameters that leads to a violation of that, e.g., two columns of J f ( s ) are identical in terms of element values and support.
So it follows that
[Equation]
Then we can construct ˜ U := (cid:104) ˜
U 1 · · · ˜ U s (cid:105) .
First, we set ˜ U
i ′ ,j ′ as a unique non-zero entry in column ˜
U j ′ .
For simplicity, we can just set ˜ U
i ′ ,j ′
= 1 .
For other column ˜
U j , where j ̸ = j ′ and ¯ U
i,j ̸ = 0 , we set ˜
U
i,j = 1 .
Therefore
[Equation]
Since Assumption iii of Thm.
2 covers all columns, Eq.
(69) holds for any j ′ ∈ { 1 , . . .
, s } .
If there are multiple columns of ¯ U with more than one nonzero entry, we derive Eq.
(69) for each of them.
We denote the set of different target column indices j ′ as J .
For j ∈ J , we set ˜
U
i j ,j = 1 , where i j is the unique index of the corresponding non-zero entry in column ˆ U j .
For other column ˜
U j , where j / ∈ J and ¯ U i,j ̸ = 0 , we set ˜
U
i,j = 1 .
Then we have
[Equation]
As noted previously, every column index j corresponds to a unique row index i .
˜
U is a permutation matrix and ˜ U ˜
U
⊤ =
I s .
It then follows that (cid:13)(cid:13)
J f ( s ) ¯ U (cid:13)(cid:13) 0 > (cid:13)(cid:13)(cid:13)
J f ( s ) ˜ U
(cid:13)(cid:13)(cid:13) 0 , which contradicts the definition of ¯ U .
A.6 Proof of Proposition
Proposition 2.
Let the observed data be sampled from a nonlinear ICA model as defined in Eqs.
(1) , (2) , and (7) .
Suppose the following assumptions hold:
i. (Independent Influences): The influence of each source on the observed variables is independent of each other, i
e., each partial derivative ∂ f /∂s
i is independent of the other sources and their influences in a non-statistical sense.
ii.
The mixing function f and its inverse are twice differentiable.
iii.
The Jacobian determinant of mixing function can be factorized as det( J f ( s ))
=
(cid:81) ni
=
y i
( s i ) , where y i is a function that depends only on s i .
Note that volume-preserving transformation is a special case when y i ( s i )
= 1 , i = 1 , . .
. , n .
iv.
During estimation, the columns of the Jacobian of the estimated unmixing function are regularized to be mutually orthogonal and with equal euclidean norm.
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Proof.
We first restate the setup here.
Consider the following data-generating process of nonlinear
ICA
[Equation]
where x denotes the observed random vector, s is the latent random vector representing the marginally independent sources, p s i is the marginal PDF of s i ,
p s is the joint PDF, and f : s → x denotes a nonlinear mixing function.
We preprocess the observational data by centering the columns of the
Jacobian of the mixing function.
This could be achieved by applying a fixed transformation on the observational data that is equivalent to left multiplying the Jacobian of the mixing function with the centering matrix U :
=
I n − (1 /n ) 11 ⊤ , where I n is the identity matrix of size n and 1 is the ndimensional vector of all ones.
After centering, according to the assumption of independent influences, the columns of the Jacobian matrix are uncorrelated and of zero-means, and thus orthogonal to each other 4 .
Since the preprocessing transformation U is fixed w.r.t.
no variables other than n , which is the fixed number of dimensions, it could always be recovered after reconstruction.
We assume that the distribution of the estimated sources ˆ s follows a factorial multivariate Gaussian:
[Equation]
where Z i > 0 is a constant.
The sufficient statistics θ i, 1 = − µ i σ
i
and θ i, 2 = 12 σ 2 i are assumed to be linearly independent.
We set the variances σ 2 i to be distinct without loss of generality, which can be placed as a constraint during the estimation process as the sufficient statistics of the estimated distribution are trainable.
Our goal here is to show that function h
= ˆ f − 1 ◦ f is a composition of a permutation and a componentwise invertible transformation of sources.
By using chain rule repeatedly, we write the Jacobian of function h :
=
ˆ f − 1 ◦ f as
[Equation]
where ˆ λ ( x ) is a scalar,
ˆ O ( x ) and O ( s ) represent the corresponding orthogonal matrices, and D ( s ) denotes the corresponding diagonal matrix.
The fourth equality follows from assumption i.
Since J h ( s ) =
ˆ λ ( x ) ˆ O
( x ) O ( s ) D ( s ) by Eq.
(75) and ˆ O
( x ) O ( s ) is an orthogonal matrix, the columns of J h ( s ) are orthogonal to each other, i.e.,
[Equation]
where J h ( s ) : ,j is the j -th column of matrix J h ( s ) .
This can be rewritten as
[Equation]
By assumption, the components of the true sources s are mutually independent, while the estimated sources ˆ s follows a multivariate Gaussian distribution:
[Equation]
[Equation]
where Z i > 0 is a constant.
The sufficient statistics θ i, 1 and θ i, 2 are assumed to be linearly independent.
Applying the change of variable rule, we have p s ( s ) =
p ˆ s (ˆ s ) | det( J h ( s ))
| , which, by plugging in Eq.
(78) and taking the logarithm on both sides, yields
[Equation]
According to assumption iii, we have the factorization det( J h ( s ))
= (cid:81) ni
=
y i
( s i ) , where y i is a function that depends only on s i .
This implies that
[Equation]
Then, based on Lemma 1 by Yang et al.
(2022), we have the following equation:
[Equation]
Specifically, Eq.
(80) is similar to Eq.
(17’) and Eq.
(17) in Yang et al.
Eq.
(81) is similar to Eq.
(20) in Yang et al.
The only difference is that there is no auxiliary variable in Eqs.
(80) and (81), which does not influence the deriviation of Lemma 1 by Yang et al. (2022).
The assumptions of the lemma are satisfied here, with the alternation that Eq.
(12) in Yang et al.
(2022) is weakened to Eq.
(17’) in Yang et al.
(2022), which does not influence the deriviation according to
Appx.
B in Yang et al.
Thus, based on Lemma 1 in (Yang et al., 2022) but without auxiliary variables, we could obtain Eq.
Then, we define the following matrices:
[Equation]
We rewrite Eq.
(81) and Eq.
(77) as
[Equation]
Because all elements in matrices defined in Eq.
(82) are positive, we can take sqaure roots on both sides of equations in Eq.
(83), yielding
[Equation]
84) where O 1 and O 2 are the corresponding orthogonal matrices ( O T 1 O 1 =
O T 2 O 2 = I n ) and Λ 1 /denotes the matrix by applying entry-wise square root to the entries of diagonal matrix Λ .
Then we rewrite Eq.
(84) as
[Equation]
By reformulating, we have
[Equation]
We consider both sides of Eq.
(86) as a singular value decomposition of the LHS.
Since σ j ̸ = σ k , ∀ j ̸ = k , elements in Λ 1 / 2 1 Λ − 1 / 2are distinct.
Therefore, non-zero elements in Λ 1 / 2 1 Λ − 1 / 2are also distinct because Λ 2 is an identity matrix.
Because the mixing function is assumed to be invertible, the Jacobian of the mixing function J f ( s ) =
O ( s ) D ( s ) is of full-rank.
Therefore, all diagonal elements in the diagonal matrix Λ 1 / 2 1 Λ − 1 / 2are non-zero.
Then, based on the uniqueness of singular value decomposition (see e.g. Proposition 4.1 in Trefethen and Bau III (1997)), O 2 is a composition of a permutation matrix and a signature matrix.
Therefore,
J h ( s ) =
Λ − 1 /
O 2 Σ (2 , s ) 1 / 2 is a generalized permutation matrix.
It thus follows that the function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
A.7 Proof of Proposition
Proposition 3.
The following inequality holds
[Equation]
) with equality iff.
J ˆ f − 1 ( x ) =
O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar.
Proof.
According to Hadamard’s inequality, we have
[Equation] [Equation]
with equality iff.
vectors ∂ ˆ f − 1 ∂x
i
i = 1 , 2 , . .
. , n are orthogonal.
Then applying the inequality of arithmetic and geometric means yields
[Equation]
with the first equality iff.
for all i = 1 , 2 , . . . , n , (cid:13)(cid:13)(cid:13) ∂ ˆ f − 1 ∂x i (cid:13)(cid:13)(cid:13) 2 is equal.
Because J ˆ f − 1 ( x ) is non-singular, ∂ ˆ f − 1 ∂x i are orthogonal and equal to each other for all i = 1 , 2 , . .
. , n iff.
J ˆ f − 1 ( x ) =
O ( x ) λ ( x ) .
B Experiments
In order to generate observational data satisfying the required assumptions, we simulate the sources and mixing process as follows:
SS .
To guarantee that the ground-truth nonlinear mixing process satisfies the structured sparsity condition (Assumption iv in Thm. 1), we generate observed variables with “structured” multi-layer perceptrons (MLPs): Each observed variables is only a nonlinear mixture of its own parents.
For example, if the observed variable x 1 has parents s 1 and s 2 , then x 1 =
MLP( s 1 , s 2 ) .
The MLP here could be replaced by any nonlinear functions.
II .
We generate the mixing functions based on Möbius transformations with scaled sources and constant volume.
According to Liouville’s results (Flanders, 1966) (also summarized in Theorem
F.2 in Gresele et al.
, the Möbius transformation guarantees the orthogonality between the columns vectors of its Jacobian matrix, which achieves uncorrelatedness after centering.
We scaled the sources while preserving volumes before Möbius transformation with distinct scalers to make sure that the generating process is not a conformal map.
We center the columns of the Jacobian as described in Appx.
A.6.
VP .
Here we describe the generating process with a factorizable Jacobian determinant but not necessarily with orthogonal columns of Jacobian.
We use a volume-preserving flow called GIN (Sorrenson et al., 2020) to generate the mixing function.
GIN is a volume-preserving version of
RealNVP (Dinh et al., 2016), which achieves volume preservation by setting the scaling function of the final component to the negative sum of previous ones.
5 We use the official implementation of
GIN (Sorrenson et al., 2020), which is part of FrEIA.
Base .
Here we describe the generating process without restrictions on having a factorizable Jacobian determinant and orthogonal columns of the Jacobian.
Following (Sorrenson et al., 2020), we use
GLOW (Kingma and Dhariwal, 2018) to generate the mixing function.
The difference between the coupling block in GLOW and GIN is that the Jacobian determinant of the former is not constrained to be one.
The implementation of GLOW is also included in the official implementation of GIN (Sorrenson et al., 2020), which is also part of FrEIA.
The ground-truth sources are sampled from a multivariate Gaussian, with zero means and variances sampled from a uniform distribution on [0 . 5 , 3] 6 .
It is worth noting that we sample sources from a single multivariate Gaussian so that all sources are marginally independent, which is different from all previous works assuming conditional independence given auxiliary variables.
Regarding the model evaluation, we use the mean correlation coefficient (MCC) between the groundtruth and recovered latent sources.
We first compute pair-wise correlation coefficients between the true sources and recovered ones.
Then we solve an assignment problem to match each recovered source to the ground truth with the highest correlation between them.
MCC is a standard metric to measure the degree of identifiability up to component-wise transformation in the literature (Hyvärinen and Morioka, 2016).
All results are of 10 trials with different random seeds.
The sample size for the synthetic datasets is 10000 .
For experiments conducted on them, the learning rate is 0 .
01 and batch size is 1000 .
The number of coupling layers for both GIN and GLOW is set as 24 .
Regarding the image dataset, we have 25000 32 × 32 images of the drawn triangle.
The statistic of the dataset is described in (Yang et al., 2022), with the difference that we only use one class of triangles for unconditional priors.
For experiments conducted on images, the learning rate is 3 × 10 −and batch size is 100 .
The number of coupling layers for the estimating method GIN is set as 10 .
The experiments are directly conducted with the official implementation of GIN (Sorrenson et al.
, 2020)with additional regularization terms and on 4 CPU cores with 16 GB RAM.