On the Identifiability of Nonlinear ICA: Sparsity and Beyond
Anonymous Author(s)
AffiliationAddress email
Abstract
Nonlinear independent component analysis (ICA) aims to recover the underlyingindependent latent sources from their observable nonlinear mixtures.
How to makethe nonlinear ICA model identifiable up to certain trivial indeterminacies is a longstanding problem in unsupervised learning.
Recent breakthroughs reformulate thestandard independence assumption of sources as conditional independence givensome auxiliary variables (e.g., class labels and/or domain/time indexes) as weaksupervision or inductive bias.
However, nonlinear ICA with unconditional priorscannot benefit from such developments.
We explore an alternative path and consideronly assumptions on the mixing process, such as Structural Sparsity or Independent
Influences .
We show that under specific instantiations of such constraints, the independent latent sources can be identified from their nonlinear mixtures up to a permutation and a component-wise transformation, thus achieving nontrivial identifiability of nonlinear ICA without auxiliary variables.
We provide estimation methodsand validate the theoretical results experimentally.
The results on image data suggest that our conditions may hold in a number of practical data generating processes.
Introduction
Nonlinear independent component analysis (ICA) is fundamental in unsupervised learning.
Itgeneralizes linear ICA (Comon, 1994) to identify latent sources from observations, which areassumed to be a nonlinear mixture of the sources.
For an observed vector x , nonlinear ICA expressesit as x = f ( s ) where f is an unknown invertible mixing function and s is a latent random vectorrepresenting the (marginally) independent sources.
The goal is to recover function f as well as sourcess from the observed mixture x up to certain indeterminacies.
While nonlinear ICA is of generalinterest in a variety of tasks, such as disentanglement (Lachapelle et al., 2022) and unsupervisedlearning (Oja, 2002), its identifiability has been a long-standing problem for decades.
The keydifficulty is that, without additional assumptions, there exist infinite ways to transform the observationsinto independent components while still mixed w.r.t.
the sources (Hyvärinen and Pajunen, 1999).
To deal with this challenge, existing works introduce the auxiliary variable u (e.g., class label,domain index, time index) and assume that sources are conditionally independent given u (Hyvärinenand Morioka, 2016, 2017; Hyvärinen et al., 2019;
Khemakhem et al., 2020; Lachapelle et al.
Most of them require the auxiliary variable to be observable, while clustering-based methods(Willetts and Paige, 2021) and those for time series (Hälvä and Hyvärinen, 2020; Hälvä et al., 2021; Yao et al., 2021) are exceptions.
These works impose mild restrictions on the mixing process butrequire many distinct values of u for identifiability, which might restrict their practical applicability.
Motivated by this, Yang et al.
(2022) reduce the number of required distinct values by strengtheningthe functional assumptions on the mixing process.
As described, all these results are based onconditional independence of the sources given auxiliary variable u , instead of the standard marginal independence assumption, to achieve identifiability in specific tasks, thanks to the weak supervisionprovided by the auxiliary variables.
The identifiability of the original setting without auxiliaryvariables, which is crucial for unsupervised learning, stays unsolved.
Instead of introducing auxiliary variables, there exists the possibility to rely on further restrictionson the mixing functions to achieve identifiability, but limited results are available in the literatureover the past two decades.
With the assumption of conformal transformation, Hyvärinen and Pajunen(1999) proved identifiability of the nonlinear ICA solution up to rotation indeterminacy for only twosources.
Another result is shown for the mixture that consists of component-wise nonlinearities addedto a linear mixture (Taleb and Jutten, 1999).
In this work, we aim to show the identifiability of nonlinear ICA with unconditional priors.
Weprove that without any kind of auxiliary variables, the latent sources can be identified from nonlinearmixtures up to a component-wise invertible transformation and a permutation under assumptionsonly on the mixing process.
Specifically, we mainly focus on the structural sparsity condition w.r.t.the structure from sources to observed variables, i.e., the support of the Jacobian matrix of the mixingfunction.
Under structural conditions on the mixing process, one could identify the true sources bysparsity regularization during estimation (Thm. 1).
By removing rotation indeterminacy, structuralsparsity benefits the identifiability of linear ICA with Gaussian sources as well, which was previouslythought to be unsolvable (Prop. 1, Thm. 2).
Besides, we also develop another identifiability theorybased on Independent Influences , which is an analogous concept of “sparsity” regarding the individualinfluencing process instead of the support (Thm. 4).
We establish, to the best of our knowledge, oneof the first identifiability results of the original nonlinear ICA model in a fully unsupervised setting.
Moreover, we also provide identifiability results for the underdetermined case, where the number of observed variables is larger than that of sources (Thm. 2, Thm. 3).
This is of great practical interest andcannot be handled with previous conditions on nonlinear ICA, even with the help of auxiliary variables(Hyvärinen et al., 2019).
We validate our theoretical claims experimentally, and the results on the image dataset suggest that our conditions appear to be reasonable for practical data generating processes.
Preliminaries
We consider the following data-generating process of ICA
[Equation]
where n denotes the number of latent sources, x = ( x 1 , .
. . , x n ) denotes the observed random vector,s =
( s 1 , . . .
, s n ) is the latent random vector representing the marginally independent sources, p s i isthe marginal probability density function (PDF) of the i -th source
s i , p s is the joint PDF of randomvector s , and f : s → x denotes a nonlinear mixing function.
For linear ICA, Eq.
(2) is restricted asx =
As , (3) where A :=
[ A : , 1 · · · A : ,n ] denotes the mixing matrix.
The goal of ICA is to learn an estimatedunmixing function ˆ f − 1 : x → ˆ s (or alternatively in the linear case, an estimated unmixing matrixˆ A − 1 ), such that ˆ s =
(ˆ s 1 , . . .
, ˆ s n ) consists of independent estimated sources.
Notations.
Throughout this work, for any matrix M , we use M i, : to refer to its i th row, and M : ,j toindicate its j th column.
For any set of indices S ⊂ { 1 , . . .
, m } × { 1 , . . . , n } , we analogously denote
S i, : := { j | ( i, j ) ∈ S} and S : ,j :=
{ i | ( i, j ) ∈ S} .
We denote by | · | the cardinality of a set.
Wealso define the following technical notations.
Definition 1.
Given a subset S ⊆ { 1 , . . . , n } , the subspace R n S is defined as
[Equation]
In other words, R n S refers to the subspace of R n indicated by an index set S .
In the following, wedefine the support of a matrix.
Definition 2.
The support of matrix M ∈ R m × n is defined as
[Equation]
With slight abuse of notation, we also reuse the notation supp to denote the support of a matrix-valuedfunction, depending on the context.
Definition 3.
The support of function M : Θ
→ R m × n is defined assupp( M (Θ)) :
= { ( i, j ) | ∃ θ ∈ Θ , M ( θ ) i,j ̸ = 0 } .
Identifiability with Structural Sparsity3.1 Nonlinear ICA with Unconditional Priors: Structural Sparsity
In this section, we consider the identifiability of nonlinear ICA with unconditional priors, where themixing function f in Eq.
(2) is nonlinear.
Different from recent breakthroughs in identifiable nonlinear
ICA with side information (Hyvärinen et al., 2019; Lachapelle et al., 2022), in our setting, sources sin Eq.
(1) do not need to be mutually conditionally independent given an auxiliary variable.
Instead,our setting is consistent with the original ICA problem based on a general marginal independenceassumption of sources.
Besides, no particular form of the source distribution is assumed, which is different from most works that directly assume an exponential family (Hyvärinen et al., 2019).
The goalof identifiable nonlinear ICA is to estimate the unmixing function[-
-] ˆ f − 1 so that the independent sourcesare identified up to a certain indeterminacy, which is usually a composition of a component-wise invertible transformation and a permutation (Hyvärinen and Pajunen, 1999).
We formulate the structuralsparsity condition below and present the identifiability result with its proof shown in Appx.
A.1.
Theorem 1.
Assume
i
The observed data is sampled from a nonlinear ICA model as defined in Eqs.
(1) and (2) .
ii Mixing function f is invertible and smooth.
Its inverse is also smooth.iii Denote by F the support of (all possible values of) the Jacobian J f ( s ) .
For all i ∈{ 1 , . . . , n } , there exist { s ( ℓ ) } |F
i, : | ℓ =1 such that span { J f ( s ( ℓ ) )
i, : } |F i, : | ℓ =1 = R n F i, : .iv
| supp( J ˆ f (ˆ s ))
| ≤ | supp( J f ( s )) | .v
(Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that (cid:92) i ∈C k supp( J f ( s ) i, : ) =
{ k } .
102 Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
Assumption iii in a generic sense rules out a set of specific parameters where the Jacobian is partiallyconstant and can be contained in a proper affine subspace aligned by its support (i.e., R n G i, : ).
Assumption iv corresponds to incorporating a suitable sparsity regularization term into the estimating processwith no restriction on the ground truth.
It helps to find the estimated mixing function with the minimal
L 0 norm among all functions that allow the model to perfectly fit the true distribution of sources.
Thus,our discussion will focus on the assumption of structural sparsity, which is the core of our theory.
We start with the intuition of the proposed direction on exploiting sparsity on the mixing processfor identifiability.
As discussed in previous works (Hyvärinen and Pajunen, 1999), there existinfinitely many ways to preserve the independence among variables after mixing them up, such as Darmois construction (Darmois, 1951) and measure-preserving automorphism (MPA) (Hyvärinenand Pajunen, 1999).
Besides, sources with non-Gaussian priors could be transferred to marginallyindependent Gaussians by trivial point-wise transformations (Hyvärinen et al., 2019).
Thus, theexisting strategy of exploiting independence and non-Gaussianity for identification fails in thenonlinear case.
However, if the true structure is sparse enough, any alternative solution that producesany indeterminacy beyond component-wise transformations and permutations (e.g., mixtures orrotations) may correspond to a denser structure.
In the light of that, instead of restricting thefunctional class (e.g., post nonlinear models (Taleb and Jutten, 1999) or conformal maps (Hyvärinenand Pajunen, 1999)) or introducing auxiliary variables for extra structural dependencies betweenlatent sources (Hyvärinen et al., 2019; Lachapelle et al., 2022), we leverage the sparsity pattern, i.e.,the support of the Jacobian of the mixing function, to identify the sources.
The structural sparsity assumption (Assumption v in Thm. 1) implies that,for every latent source s i , there exists a set of observed variable(s) suchthat s i is the only latent source that participates in the generation of allobserved variable(s) in the set.
Graphically, for every latent source s i , thereexists a set of observed variable(s) such that the intersection of their/itsparent(s) is s i (e.g., for s 1 in Fig.
1, there exist x 1 and x 4 such that theintersection of their parents is s 1 .).
This encourages the connections to besparse enough to make the disentanglement of each source based on structural conditions possible.
Together with the sparsity regularization duringestimation (Assumption iv ) and other mild conditions, Assumption v illustrates a graphical pattern of nonlinear ICA models that could be identifiedby exploiting structural sparsity.
It indicates a structural criterion of thesparsity that is needed for identifiability.
Beyond that, structural sparsityis also of practical interest for the interpretability of results (Zhang et al.
,2009) and identification of other latent variable models (Bing et al., 2020;
Rohe and Zeng, 2020; Moran et al., 2021; Rhodes and Lee, 2021). Meanwhile, various versions of Occam’s razor (e.g., faithfulness (Spirtes et al., 2000), minimality principle (Zhang, 2013), and frugality(Forster et al., 2020)) are also fundamental to the identifiability of the underlying causal structure.
With structural sparsity, Thm. 1 shows the identifiability of nonlinear ICA while maintaining thestandard mutually marginal independence assumption of sources.
This is consistent with the originalsetting of ICA and plays an important role in a more general range of unsupervised learning tasks,compared to previous works relying on conditional independence given auxiliary variables.
Moreover,one may modify Thm.
1 to consider the generating process based on auxiliary variables by imposingsimilar restrictions on the mixing process.
Different from previous works (Hyvärinen et al., 2019;
Yang et al., 2022), this extension removes common restrictions on the required auxiliary variables,such as the number of distinct values (e.g., class labels), non-proportional variances, or sufficientvariability.
Therefore, although our main result (Thm. 1) focuses on nonlinear ICA with unconditionalpriors, it also provides potential insight into improving the flexibility of utilizing the auxiliary variablewhen it is available.
As a trade-off, the additional assumptions on the sparsity might limit its usage.
Besides, under additional assumptions, we could reduce the indeterminacy in Thm. 1 and furtheridentify sources up to a component-wise linear transformation.
Most works in nonlinear ICA focuson the identifiability up to a component-wise invertible transformation and a permutation (Hyvärinenand Morioka, 2016; Khemakhem et al., 2020), which is also called permutation-identifiability in theliterature of disentanglement (Lachapelle et al., 2022).
This indeterminacy is trivial compared tothe fundamental nonuniqueness of nonlinear ICA and is analogous to the indeterminacy involvingrescaling and permutation in linear ICA (Hyvärinen and Pajunen, 1999).
Recently, Yang et al.
(2022)provided an identifiability result based on auxiliary variables that further reduces the indeterminacyof the component-wise nonlinear transformation to a linear one.
Inspired by it but without auxiliaryvariables, we show conditions for the identifiability of nonlinear ICA with unconditional priors up toa component-wise linear transformation, with the proof in Appx.
A.2.
The reduced indeterminacymay give rise to a more informative disentanglement and open up the possibility for further improvingthe quality of recovery with only element-wise linear operations, such as the Hadamard product.
Corollary 1.
Assume i The observed data is sampled from a nonlinear ICA model as defined in Eqs
(1) and (2) .
ii
The function h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation
and a permutation.
iii
The mixing function f is volume-preserving.
iv The source distribution p s ( s ) is a factorial multivariate Gaussian.
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise linear transformation and a permutation.
3.2 Removing
Rotation Indeterminacy with Structural Sparsity
Rotation indeterminacy is one of the major obstacles to the identifiability of ICA.
Linear ICA exploitsthe maximization of the non-Gaussianity of the estimated sources (e.g., Kurtois (Hyvärinen and Oja,
As a direct result, the typical assumption is that at most one of the sources can have Gaussiandistribution.
In contrast, for the nonlinear case, a trivial point-wise function could transform sourcesto have any marginal distribution including Gaussian thus invalidating the effect of non-Gaussianity.
Therefore, removing rotation indeterminacy remains critical for the identifiability in the nonlinearcase.
For instance, “rotated-Gaussian” MPA produces nonuniqueness in nonlinear ICA due to rotationindeterminacy (Hyvärinen and Pajunen, 1999).
It first maps the distribution to an isotropic Gaussian,then applies a rotation and maps it back without affecting its original distribution.
To address itfor identifiability, Yang et al.
(2022) assume that there do not exist two conditionally independent
Gaussians such that their variances are proportional.
In the previous section, we have introduced the structural sparsity for the identifiability of nonlinear
ICA.
One may wonder whether it straightforwardly leads to the identifiability of linear ICA with Gaussian sources (Gaussian ICA), which was previously perceived to be impossible due to its rotation indeterminacy.
The idea is simple–any rotation of the true mixing matrix A will be less sparse if A satisfiesthe proposed structural sparsity assumption.
While the joint distribution of Gaussian sources staysinvariant across rotations, the sparsity of the mixing matrix (i.e., L 0 norm) keeps changing.
We firstconsider Assumption v in Thm.
1 for Gaussian ICA as an extension.
Because the Jacobian of linearmixing function is fixed, Assumption iii of Thm.
1 does not directly hold in the linear Gaussian case.
As a result, we cannot directly apply Thm.
1 here
and therefore propose some alternative conditions:
Proposition 1.
Assume
i
The observed data is sampled from a linear ICA model defined in Eqs.
(1) and (3) with Gaussian sources.
ii
Mixing matrix A is invertible.
iii There exists a matrix ˆ A such that ∀ j ∈ supp
( A ) i, : , supp( ˆ AA − 1 ) j, : ∈ R n supp(ˆ A ) i, : .iv
| supp( ˆ A ) | ≤ | supp( A ) | .
v (Structural Sparsity)
For all k ∈ { 1 , . . . , n } , there exists C k such that
[Equation]
A i
[Equation]
) = { k } .
200 Then ˆ A =
ADP , where D is a diagonal matrix and P is a column permutation matrix.
The proof is provided in Appx.
A.3.
Similar to Assumption iii of Thm. 1, Assumption iii of Prop.
1excludes a specific set of parameters that makes structure-based identification ill-posed.
The underdetermined case.
For the full identifiability of nonlinear ICA, it is noteworthy that theinvertibility of the mixing function is technically essential in our result (Thm. 1).
This invertibilityimplies that the number of observed variables is equal to that of sources.
Although that setting israther common in the literature, especially for the theory of nonlinear cases (Hyvärinen and Pajunen,1999; Hyvärinen et al., 2019), underdetermined ICA (i.e., there are more observed variables), maybe of more practical interest and more challenging (Kim and Yoo, 2004).
Therefore, we develop analternative structural sparsity condition for the underdetermined case.
As a trade-off, in the nonlinearcase, the structural condition that removes the requirement of invertibility could only deal with therotation indeterminacy.
Thus we start from Gaussian ICA to introduce the condition.
Definition 4.
Given a matrix S ∈ R m × n , we define the function overlap : R m × n → { 0 , 1 } m × n as (overlap( S )) ij = (cid:26)if S ij =
and it is not the only nonzero entry in S i, : otherwise.
Theorem 2.
Assume
i
The observed data is sampled from a linear ICA model defined in Eqs.
and (3) with Gaussian sources.
Differently, the number of observed variables (denoted as m ) could be larger than that of the sources n , i.e., m ≥ n .
ii
The nonzero coefficients of matrix A are randomly drawn from a distribution that is abso- lutely continuous with respect to Lebesgue measure.
iii | supp( ˆ A ) | ≤ | supp( A ) | .iv
(Structural Sparsity)
Given C ⊆ { 1 , 2 , . . .
, n } where |C| > 1 , let A C ∈ R m ×|C| represents a submatrix of the mixing matrix A ∈ R m × n consisting of columns with indices C .
Then, for all k ∈ C , we have
[Equation]
220 Then ˆ A =
ADP , where D is a diagonal matrix and P is a column permutation matrix.
The proof is provided in Appx.
A.4.
Intuitively, Assumption iv in Thm.
2encourages the variability between the influences of each source.
For eachsource, its influence on the observations should be as distinctive as possible,which is correlated with the sparsity of the mixing matrix.
Fig.
2 is an illustration of the assumption.
In that case, k = 1 and C = { 1 , 2 , 3 } .
Let A Cdenotes the submatrix shown in the figure, where (cid:12)(cid:12)(cid:12)(cid:12) (cid:83)
[Equation]
= 7 .
Bluedotted square denotes overlap( A C ) and we have rank(overlap( A C ))
= 2 . Thus,(cid:12)(cid:12)(cid:12)(cid:12)
(cid:83) k ′ ∈C supp( A k ′ ) (cid:12)(cid:12)(cid:12)(cid:12) − rank(overlap A C ))
=
5 (the number of black dots), whichis larger than | supp( A 1 ) | = 4 .
For the nonlinear case, we prove that it couldhelp to distinguish spurious solutions due to the rotation indeterminacy, i.e., the“rotated-Gaussian” MPA (Defn. 2.5 in Gresele et al.
Gresele et al.
(2021) prove that this classof MPAs could be ruled out with assumptions of conformal maps, non-Gaussianity and orthogonalityof the Jacobian of the mixing function.
Differently, we address it with the proposed structural sparsitycondition (Assumption iv in Thm. 2).
The corresponding theorem for the nonlinear case, with itsproof given in Appx.
A.5, is as follows:
Theorem 3.
Given a nonlinear ICA model defined in Eqs.
(1) and (2) , where f is the true mixingfunction.
Consider ˆ f = f ◦ G − 1 ◦
U ◦ G , where G denotes an invertible Gaussianization 1 thatmaps the distribution to an standard isotropic (rotation-invariant) Gaussian, U denotes a rotation,and G − 1 maps the distribution back to that before applying ˆ U ◦ G .
If Assumptions ii, iii and iv of
Thm. 2 are satisfied by replacing A with J f ( s ) and ˆ A with J ˆ f ( s ) , then function h :
= ˆ f − 1 ◦ f is acomposition of a component-wise invertible transformation and a permutation.
Besides removing rotation indeterminacy, Thm. 3 also provides an extra insight for the full identifiability of nonlinear ICA, because all previous works assume that the nonlinear mixing function must beinvertible (Hyvärinen and Morioka, 2016; Yang et al., 2022) even with the help of auxiliary variables.
Identifiability with Independent Influences
In the previous section, we have presented theoretical results based on conditions of structural sparsity.
Alternatively, we show in this section that sparsity conditions on the support might not be necessary insome specific scenarios, and that one can leverage the independence (in a non-statistical sense) amongthe influences from different sources to the observations to achieve identifiability of nonlinear ICA.
This implies sparse interactions between the individual influencing processes and could be viewed asan analogous concept of “sparsity” w.r.t.
influences.
We show that without any kind of auxiliary variables, the full identifiability of nonlinear ICA can also be provided given the condition of independentinfluences, thus complementing the results shown above from a different view of “sparsity”.
Nonlinear ICA with unconditional priors: Independent Influences.
We now specify the settingof the required estimating process.
We sample the estimated sources ˆ s from a multivariate Gaussian:
[Equation]
where Z i > 0 is a constant.
The sufficient statistics θ i, 1 = − µ i σ
i
and θ i, 2 = 12 σ 2 i are assumed to belinearly independent.
We constraint the variances σ 2 i to be distinct without loss of generality.
Theorem 4.
Consider the model defined in Eqs.
(1) , (2) , and (7) .
Assumei (Independent Influences): The influence of each source on the observed variables is independent of each other, i.e., each partial derivative ∂
f /∂s i is independent of the other sourcesand their influences in a non-statistical sense.
ii
The mixing function f and its inverse are twice differentiable.
iii
The Jacobian determinant of mixing function can be factorized as det( J f ( s ))
=
(cid:81) ni
=
y i
( s i ) ,where y i is a function that depends only on s i .
Note that volume-preserving transformationis a special case when y i ( s i )
= 1 , i = 1 , . .
. , n .
iv During estimation, the columns of the Jacobian of the estimated unmixing function are
regularized to be mutually orthogonal and with equal euclidean norm
Then h :
= ˆ f − 1 ◦ f is a composition of a component-wise invertible transformation and a permutation.
The proof is provided in Appx.
A.6.
Note that some part of the proof technique is inspired by orbased on Lemma 1 in Yang et al.
Assumption iv is a regularization during estimation and thusputs no restriction on the true generating process.
We achieve it by optimizing an objective functionwith a designed regularization term, of which the property is as follows (proof in Appx.
A.7):
Proposition 2.
The following inequality holds
[Equation]
with equality iff.
J ˆ f − 1 ( x ) =
O ( x ) λ ( x ) , where O ( x ) is an orthogonal matrix and λ ( x ) is a scalar.
Regarding the assumption of independent influences, to build intuition, one can consider a filmmakingprocess in a safari park, of which the main characters are wild animals.
The task of the boom operatorshere is to record the sound of animals.
As there are many animals and microphones, the mixing(recording) process is highly dependent on the relative positions between them.
Thus, the animal(source s i ), loosely speaking, influences the recording (mixing function f ) through its position.
Themoving direction and speed of them can be loosely interpreted as the partial derivative w.r.t.
thei -th source, i.e. ∂ f /∂s i .
Assuming that the wild animals are not cooperative enough to fine-tunetheir positions and speeds for a better recording and the safari park is not crowded, the influences ofanimals on the recording are generated independently by them in the sense that they are not affectedby the others while moving.
As a result of that generating process, the column vectors of the Jacobianof the mixing function are uncorrelated with each other and the partial derivative w.r.t.
to each sourceis independent of other sources.
Other practical scenarios include fields that adopt independentinfluences in process like orthogonal coordinate transformations (Gresele et al., 2021), such asdynamic control (Mistry et al., 2010) and structural geology (De Paor, 1983).
It is worth noting thatthis assumption of independent influences is different from the orthogonality condition proposed in(Gresele et al., 2021).
For instance, if the sample mean is not zero, uncorrelatedness differs fromorthogonality.
And we argue that uncorrelatedness as a specific way to achieve independence mightbe more sensible as a condition on the influences.
Besides, in (Gresele et al., 2021), orthogonality,together with conformal map and the others, are proposed as conditions to rule out two specific typesof spurious solutions (i.e., Darmois construction (Darmois, 1951) and “rotated-Gaussian” MPAs(Locatello
et al., 2019)) without an identifiability result, although its empirical results are encouraging.
Having said that, a violation of the assumption of independent influences could be ascribed to adeliberate global adjustment of sources.
For example, cinema audio systems are carefully adjusted toachieve a homogeneous sound effect on every audience.
The position and orientation of each speakerare fine-tuned according to the others.
In this case, it may be difficult to distinguish the influencesfrom different speakers because of the fine-tuning.
This may lead to a high degree of multicollinearityacross the columns of Jacobian, thus violating the assumption of independent influences.
Regarding Assumption iii in Thm. 3, we first note that mixing functions with factorial Jacobiandeterminants are of a much wider range as compared to component-wise transformations.
To see this,
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(a) SS .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(b) II .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(c) VP .
4 Ground-truth Sources    4 R e c o v e r e d S o u
r
c
e s
Correlation Matrix 0.
(d) Base .
a straightforward example is the volume-preserving transformation, which has been widely adopted ingenerative models (Zhang et al., 2021).
In fact, all transformations with constant Jacobian determinantcan be factorized w.r.t.
the sources, which, generally speaking, are not component-wise.
Besides,consistent with the empirical results in Yang et al.
(2022), there exists other non-volume-preservingtransformations with factorial Jacobian determinant 2 .
Moreover, the volume-preserving assumptionhas been demonstrated to be helpful to weaken the requirement of auxiliary variables, specifically byreducing the number of required labels (Sorrenson et al., 2020; Yang et al., 2022).
Thm. [-
 takesthis further-]4 takes thisfurther by imposing a weaker version, i.e., Assumption iii , which, together with the other constraints,removes the need for any auxiliary variable.
Experiments
To validate the proposed theory of the identifiability of nonlinear ICA with unconditional priors, weconduct experiments based on our main condition, i.e., structural sparsity (Thm. 1), as well as thecondition of the independent influences (Thm. 4).
Setup.
For the required regularization during estimation, we consider a regularized maximumlikelihood approach with the following objective: L (ˆ f − 1 ; x ) =
E x (cid:104) log p ˆ f − 1 ( x ) − λ R
(ˆ f − 1 , x ) (cid:105) ,where λ is a regularization parameter and R (ˆ f − 1 , x ) is the regularization term.
For Thm. 1, we use
L 1 norm as an approximation of the L 0 norm for efficiency (Donoho and Elad, 2003), therefore
R (ˆ f − 1 , x ) := ∥
J ˆ f − 1 ( x ) ∥ 1 ; for Thm. 4, R (ˆ f − 1 , x ) corresponds to LHS of Eq.
Following(Sorrenson et al., 2020; Yang et al., 2022), we train a GIN (Sorrenson et al., 2020) to maximize thethe objective function L (ˆ f − 1 ; x ) .
[Figure]
Ablation study.
We conduct an ablation study to verify thenecessity of the proposed assumptions.
Specifically, we focus onthe following models that correspond to different assumptions:
(SS)
The assumption of structural sparsity, as well as otherassumptions in Thm. 1, are satisfied; (II)
The assumption ofindependent influences, as well as other assumptions in Thm.4, are satisfied; (VP)
Compared to II , only the assumption ofindependent influences is violated while the other assumptions
(e.g., factorizable Jacobian determinant) are still satisfied; (Base) The vanilla baseline.
Compared to VP , the (un)mixing function
is not restricted to having factorizable Jacobian determinants
The data are generated according to the required assumptions.
We also conduct comparison between the assumption of independent influences and orthogonality in(Gresele et al., 2021), which are presented in Appx.
B together with experimental settings.
Results foreach model are summarised in Fig.
For evaluation, we use the mean correlation coefficient (MCC)between the true sources and the estimated ones (Hyvärinen and Morioka, 2016).
One could observethat when the proposed assumptions are fully satisfied ( SS or II ), our model achieves the highest
MCC on average.
This indicates that it is actually possible to identify sources from highly nonlinearmixtures up to trivial indeterminacies only based on restrictions on the mixing process.
Moreover,regarding Thm. 4, the importance of Assumption iii is supported by the higher performance of VPcompared to that of Base .
The visualization of the Pearson correlation matrices is shown in Fig.
[Equation]
are constants.
[Figure] [Figure] [Figure] [Figure]
Stability.
To study the stability of the performance of identification w.r.t.
different datasetsvarying the number of sources n .
We test themodel SS (Thm. 1) with different n .
Visually, we find that SS consistently outperforms
Base (Fig. 5).
Meanwhile, when the numberof sources increases, one could observe that the
MCC of Base decreases while that of SS staysstable.
The stable empirical performance further validates our theoretical claims about theidentifiability with structural sparsity.
To study how reasonablethe proposed theories are w.r.t.
the practicalgenerating process of observational data, weconduct experiments on the “Triangles" imagedataset (Yang et al., 2022).
The process ofgenerating this dataset mimics the processof drawing triangles by humans: i)
First, wesample the elements needed for humans todraw a monochrome triangle (i.e., rotation,width, height and grey level) from a factorialmultivariate Gaussian distribution.
Differentfrom (Yang et al., 2022), we always samplefrom a single distribution in order to guaranteethat all priors are unconditionally independent;ii) Then, for each pixel, we decide whether itlocates inside the triangle based on the sampledelements and assign its gray level accordingly.
Therefore, the process is similar to human drawingtriangles.
Even though each image is generated from these semantic elements, the true generatingprocess and sources are still unknown (e.g., a pixel could be (indirectly) influenced by multipleelements in a complicated way).
We apply GIN with sparsity regularization as the estimating method.
The visualization of the identified sources (Fig. 6) indicates that our conditions may hold in practice.
Discussion
We provide identifiability results for nonlinear ICA with unconditional priors, which serve as oneof the first steps to solve a long-standing problem in unsupervised learning.
In particular, we provethat the i.i.d.
latent sources can be recovered up to a component-wise invertible transformationand a permutation with only conditions on the nonlinear mixing process (e.g., structural sparsity).
Therefore it stays closer to the original notion of ICA that is based on the marginal independenceassumption of latent sources, while previous works rely on conditional independence on auxiliaryvariables as weak supervision or inductive bias.
Besides, by removing rotation indeterminacy,structural sparsity benefits the identifiability of Gaussian ICA as well, which was also thought to beunsolvable before.
Moreover, the results on the underdetermined case are of great practical interestand introduce insight for extending identifiable nonlinear ICA to general real-world settings.
Our results on images illustrate the validity of the proposed conditions in practical data generatingprocesses.
In spite of this, it is possible that part of them is violated in several specific scenariosas discussed before.
For example, the structural sparsity conditions do not apply to fully-connectedstructures, though the practical significance of such cases may be compromised by the lack ofinterpretability.
As a complementary solution, we formulate the independent influences condition,which does not rely on the sparse structure of supports.
While arguably natural in general, it couldstill be violated due to a deliberate global adjustment of sources for homogeneity.
We argue thatthis is inevitably a trade-off between introducing auxiliary variables and imposing restrictions onthe mixing process to achieve the identifiability, whose practical use depends on the scenario andinformation available.
Future work includes further generalizing and validating our theory.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix
B. Our experiments are conducted with the official implementation of GIN (URL
provided).
All experimental settings are presented.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]