Provable Subspace Identification Under Post-Nonlinear Mixtures
Abstract
Unsupervised mixture learning (UML) aims at identifying linearly or nonlinearlymixed latent components in a blind manner.
UML is known to be challenging:
Even learning linear mixtures requires highly nontrivial analytical tools, e.g.,independent component analysis or nonnegative matrix factorization.
In this work,the post-nonlinear (PNL) mixture model is revisited, where unknown element-wisenonlinear functions are imposed after a linear mixture, making the identificationproblem even more ill-posed.
The PNL model is widely employed in differentfields ranging from brain signal classification, speech separation, remote sensing,to causal discovery.
Existing works often assume different properties on the latentcomponents (e.g., statistical independence or probability-simplex structures) toidentify and remove the unknown nonlinear functions.
This work shows that theexistence of a nontrivial null space associated with the underlying mixing systemsuffices to guarantee identification/removal of the unknown nonlinearity.
Comparedto the existing works, this finding largely relaxes the model identification conditionsof PNL models.
Consequently, a simple learning criterion is proposed that couldbenefit applications where no strong structural information on the latent componentsis known.
A finite-sample analysis is offered to characterize the performance ofthe proposed approach under realistic settings.
For implementation, we design anoptimization strategy that features a block coordinate descent algorithm.
A seriesof numerical experiments corroborate our theoretical claims.
Introduction
Unsupervised mixture learning (UML) has been broadly used in machine learning tasks, for recoveringmixed latent components or extracting representative information.
Central to the UML problemis the model identifiability problem, which answers the question if the latent components can beidentified with guarantees and inconsequential ambiguities—without using any label or trainingdata.
This is a challenging problem due to the unsupervised nature.
To address the identifiabilitychallenge, linear mixture model identification problems were tackled via exploiting properties ofthe latent components, e.g., statistical independence [1], nonnegativity [2,3], quasi-stationarity [4],boundedness [5].
This leads to a cohort of blind source separation (BSS) methods.
However, linear mixture models oftentimes fail to to perform well on real-world data, due to theunknown nonlinear distortions arising in data generation and acquisition.
In the past two decades,nonlinear mixture models have attracted unprecedentedly more attentions [6–12].
However, it isshown in [13] that the nonlinear mixture models are in general not identifiable, even with strongassumptions, e.g., statistical independence among the latent components.
To establish identifability, aline of work exploited observable auxiliary variables or specific priors (i.e., extra information aboutthe data or the latent components on top of statistical independence); see, e.g., [11,12,14,15].
Anotherline of methods take an alternative route via utilizing the structures of the nonlinear distortions.
In particular, the post-nonlinear (PNL) model [6,7,16] is a reasonable extension of the classical linearmodel by assuming unknown nonlinear transformations individually imposed on the outputs ofa underlying linear mixture model.
This structured nonlinear mixture model is used in a widerange of applications, e.g. hyperspectral imaging [17], brain data classification [18–20], and causaldiscovery [10,21,22].
In order to identify the generative model/latent components under the PNLmodel, a key step is to remove the nonlinear distortions in an unsupervised manner.
To this end, theexisting methods often explicitly rely on some properties of the latent components, e.g., statisticalindependence [6,7,16], availability of multiview data [23] and the sum-to-one and nonnegativity (orprobability-simplex) structure of the latent components [24,25].
These structures stem from specificapplications, and thus are meaningful—yet not always available.
A natural question is whetherthe nonlinear distortions are still removable if the widely used structural assumptions (e.g., thosein [6,7,16,23–25]) do not hold?
In this work, we offer a positive answer to the above inquiry—and build a new learning algorithmbased on our analyses.
Our detailed contributions are as follows.
Subspace-Based PNL Model Identification.
We propose a new nonlinearity identification andremoval criterion under the PNL model.
We show that, as long as the underlying linear mixing systemof the PNL model admits a nontrivial orthogonal complement (or, its transpose has a nontrivial nullspace), the unknown nonlinear transformations are identifiable and removable, under mild conditions.
Notably, our approach does not use any specific structural assumptions of the latent components.
Ourfinding presents a fairly different result compared to existing nonlinear mixture learning techniques.
The latter often heavily rely on the properties of the latent components.
Hence, our result may offernew insights and a useful alternative to this long-standing challenge.
An immediate consequence isthat our method can work with various types of latent components, including those not covered byexisting PNL model identification approaches in [6,7,16,23–25].
Performance Analysis Under Realistic Settings.
The model identification analysis is conductedunder the population case where unlimited data is available, with the assumption that a universalfunction learner can be used to cancel any nonlinear distortion.
To advance the understanding undermore realistic settings, we also consider the realistic scenario with finite sample and non-universalfunction learners.
The analysis is a combination or statistical generalization theory and numericaldifferentiation.
The analytical strategy is reminiscent of the recently developed framework in [23,26],with nontrivial derivations to accommodate our learning criterion.
Optimization Algorithm Design.
Based on the theoretical analysis, we formulate the identificationproblem as a constrained optimization.
The problem is challenging, as it involves learning unknownnonlinear functions represented by neural networks (NNs) under nonconvex constraints.
We proposea block coordinate descent (BCD) algorithm that alternates between two subproblems, namely, nullspace basis learning and neural network based nonlinear function learning.
This way, the latter isdisentangled with constraints, and thus off-the-shelf NN training techniques (e.g., backpropagationand Adam [27]) can be readily employed to handle the subproblem.
A series of experiments areconducted for validating our theoretical claims.
Notations.
The notations used are summarized as follows.
x, x , X represent a scalar, vector, andmatrix, respectively; ∥ x ∥ 0 ,
∥ x ∥ 1 and ∥ x ∥ 2 denote the ℓ 0 norm, ℓ 1 norm and the Euclidean norm,respectively; ⊙ and ⊛ denotes the Khatri-Rao product and the Hadamard product, respectively; fis used to denote a function f ( · ) :
R → R ; f ′ , f ′′ and f ( n ) denote the first-order, second-order andn th order derivatives of the function f , respectively; f ◦ g denotes the function composition; N ( X )means the null space of X .
Background
2.1 UML and Model Identification
In UML, the classical linear mixture model is defined as follows [28]:
[Equation]
where x ℓ ∈ R M is the ℓ th observation sample, s ℓ ∈ R K contains the K latent components, and
A ∈ R M × K is the mixing system.
The latent components of linear mixture model in (1) are generallynot identifiable, if A is not known [28].
As a remedy, information about A and/or s ℓ is often used to establish identifiability.
For instance, statistical independence among [ s ℓ ] i ’s is used in independentcomponent analysis (ICA) [1], and the nonnegativity of A ∈ R M × K + and S =
[ s 1 , · · · , s N ] ∈ R K × N +are used in nonnegative matrix factorization (NMF)
[2,3]; also see [4,5,29–31] for more conditionsthat can assist linear mixture identification.
The linear mixture model in (1) is often over-simplified for real applications.
Nonlinear effects arewidely observed in data generation and acquisition.
In many cases, the following nonlinear mixturemodel may be of more interest [32]:
[Equation]
where g : R K → R M is a structure-free invertible nonlinear function.
However, it is shown in [32]that the model in (2) is not identifiable, even if the latent components are assumed to be statisticallyindependent.
A workaround is to use more information of or additional conditions on the latentcomponents, e.g., the availability of auxiliary variables [12] exponential family priors [14], andthe slow changing property [11].
Another approach to establish model identifiability of nonlinearmixtures is by exploiting structural assumptions of the nonlinear mixing process.
This leads to thesuite of PNL mixture model learning works (see, e.g., [6,7,16,33]), which will be our focus.
2.2 Post-Nonlinear Mixture Model
The PNL mixture model is a natural extension to the linear mixture model [6,7,16,23,24,33]:
[Equation]
where g =
[ g 1 ( · ) , · · · , g M ( · )] ⊤ with each g m ( · ) :
R → R being an invertible nonlinear function.
Weassume s ℓ ∼ p ( s ) and the support of p ( s ) is a continuous and open domain S .
Although the PNL model is not the most general one, it has a myriad of of applications across differentfields, e.g., causal discovery [10, 21, 22], hyperspectral imaging [17] and brain data classification[18–20].
It is considered as an appropriate model when the unknown nonlinear distortions happenat the sensor end in data acquisition.
It also offers a valuable generalization to the linear mixturegenerative models in data analytics, e.g., those used in NMF and data clustering [3].
An important remark is that the challenge of UML under the PNL model boils down to identifyingand removing g ( · ) , as this will lead to a well understood linear mixture model in (1).
Then, anylinear mixture model identification approach in [5,28,29,31,34] can be applied to identify the latentcomponents s ℓ for ℓ = 1 , . .
. , N .
Existing works tackle the g -identification and removal problem from different angles.
In [6,16], thestatistical independent among the latent components (i.e., [ s ℓ ] k for k = 1 , . . .
, K ) is used.
Recently,[24,25] proposed to identify the PNL model with latent components satisfying the probability simplexconstraint, i.e., 1 ⊤ s ℓ = 1 , s ℓ ≥ 0 , that often arises in soft clustering problems.
Beyond that, [23]shows g ( · ) can be provably removed with the assistance of multiple views of data, where identicalcopies of s ℓ appears in the latent space of more than one data domain.
These methods offeredinsightful and useful approaches to remove g ( · ) , but the use of specific structural assumptions on thelatent components make their applicability relatively narrow.
Unlike prior works, we propose to pin down nonlinear distortions by simply utilizing a subspacestructure, instead of resorting to any specific structure of s ℓ .
Our method starts with a rudimentaryobservation:
Note that if the mixing system A is a tall matrix, N ( A ⊤ ) is a nontrivial subspace.
Thisimplies that if only the linear mixture part is considered (i.e., if g ( · ) can be removed completely),there exists vector v such that
[Equation]
where the nonzero v is a vector from N ( A ⊤ ) .
In the next section, we will show how this simpleequation (4) allows for constructing a provable g -identification and removal criterion.
Problem Formulation
In this section, we propose a learning criterion to identify/remove the nonlinear transformations g ( · ) .
Here, we consider the population case where infinite data points are available as x ℓ ∼ p ( x ) over thecontinuous support X of p ( x ) .
Under this assumption, the (idealistic) formulation is as follows: find Q , f ( · ) =
[ f 1 ( · ) , · · · , f M ( · )] ⊤ (5a)
[Equation]
(5b)
[Equation]
[Equation]
5d
[Equation]
where Q ∈ R M × D , D is the dimension of the null space (i.e., N ( A ⊤ ) ) that we hope to find with
D ≥ 1 and D < M .
The ℓ 0 -norm constraint in (5d) requires Q to be a dense matrix, which isimportant to establishing identifiability.
This will become clearer in Sec. 4.
In the formulation,f m ( · ) is a nonlinear invertible function that we look for.
Ideally, we hope to learn f m ( · ) =
g − 1 m ( · )which cancels the nonlinear transformation.
Then, Q forms a basis of N ( A ⊤ ) .
Note that underthe orthogonality constraint in (5d), a possible trivial solution with Q = 0 is avoided.
Besides, theinvertibility constraint in 5c is used to prevent other degenerate solutions, e.g., f ( x ℓ )
=
c for all ℓwith a constant c .
Provable Nonlinearity Removal
In this section, we will show that by solving (5), it is guaranteed that the nonlinear transformationg ( · ) is removed.
First, we give the following condition which will be useful in the proof.
Definition 1 (Locally Free Variables) Consider real-valued random components v =[ v 1 , . .
. , v d ] ⊤ ∈ R d , where v i resides in a continuous and open set V i ⊆ R .
Assume thatthe following holds for all jp ( v j | v − j ) > 0 , with continuous support V j | v − j given any v − j , where v − j denotes the vector with all entries but v j .
Then, the components in v are all locally freevariables.
Note that the definition above specifies that each v j has some degree of freedom no matter what valuethe v − j part is.
As a special case, independent random variable v clearly satisfies the definition offree variable.
However, statistical independence is a much stronger condition.
Dependent variablescould also be free variables.
For instance, the vectors that are subject to the probability simplex, i.e.,
[Equation]
also share this property for any v − k , which means any K − 1 of the K components in v are freevariables.
In particular, with v i and v j being free variables, we could observe any possible value ofv j given a fixed ¯ v i .
In particular, consider the functional equation q ⊤ h ( Av ) = 0 that holds for all v .
Then, we can always observe
[Equation]
with fixed ¯ v − j and v j is from certain continuous support with p ( v j | ¯ v − j ) > 0 .
This implies we maytake derivative w.r.t.
v j on the equation above, which gives us
[Equation]
as all the terms ¯ v i v j = 0 for any
i ̸ = j .
This property will help us show the identifiability result next.
Given the conditions, we have the following theorem for identifiability of the PNL model.
Theorem 1 (Nonlinearity Identification and Removal)
Under the model in (3) , assume that thecriterion (5) is solved.
In addition, assume that A ∈ R M × K is drawn from any joint absolutelycontinuous distribution, (cid:101) K of the K components of s are free variables (cf.
Def. 1), and that thelearned (cid:98) h m = (cid:98) f m ◦ g m is twice differentiable for all m ∈
[ M ] .
Suppose that
[Equation]
Then, almost surely, any (cid:98) f ( · ) that is a solution of (5) satisfies (cid:98) h m ( x ) = (cid:98) f m ◦ g m ( x ) =
c m x + d m ,∀ m ∈
[ M ] , where c m ̸ = 0 and d m is a constant, in the limit of infinite samples.
Note that Theorem 1 indicates that if any solution (cid:98) f ( · ) of (5) is found, then the composition(cid:98) h m = (cid:98) f
m ◦ g m is an affine function for all m .
We do not require all the K latent components to befree variables as long as the condition (cid:101) K ( (cid:101) K +1) ≥ M is satisfied.
Intuitively, if (cid:101) K is too small then itmeans that the remaining K − (cid:101) K components do not provide useful information, while a larger (cid:101) Kimplies that there is more diversity among the latent components to assist nonlinearity removal.
Notethat the existence of the solution implies that the null space N ( A ⊤ ) must exist ( D ≥ 1 ), i.e., M > Kshould hold.
Otherwise, one may fail to learn a nonzero Q .
On the other hand, if M is overly large,i.e., M > (cid:101) K ( (cid:101) K +1) , the model is not identifiable since there might be lack of information (i.e., (cid:101)
K notlarge enough) to pin down all the g i ( · ) ’s with i = 1 , · · · , M .
We show the sketch of proof as follows.
Proof Sketch :
By solving (5), we have the following equation for all s ℓ ∈ S
[Equation]
where h
= f ◦ g is the function composition.
First, we show the existence of solution to (5).
It is obvious that a solution is f = g − 1 , columns of
Q being an orthogonal basis of N ( A ⊤ ) and a dense matrix.
Next, we show that such a solution isunique up to only affine ambiguities.
By the assumptions, there are (cid:101) K of the K components of s are free variables.
Thus, by taking thesecond-order (cross) derivatives of the equation above w.r.t.
each of the (cid:101) K variables, as shown in (8),we have the following linear system
[Equation]
where a k is the k th column of the A matrix which corresponds to one of the free variables (withoutloss of generality, we assume that the first (cid:101) K variables of s are locally free variables).
It is straightforward that if one can show (cid:98) h ′′ m ([ As ] m ) = 0 for any m over any s , then it implies thatany (cid:98) h m ( · ) is an affine function.
Following this thought, we actually need to show that Q ⊤ ⊙ (cid:98) A isfull column rank with Q ∈ R M × D and (cid:98)
A ∈ R (cid:102) K
( (cid:102) K +
[Equation]
× M .
By Lemma 1 of [35], for matrices U ∈ R a × m and V ∈ R b × m , U ⊙ V has full column rank ifkrank( U ) + krank( V ) ≥ m + 1 with krank( U ) and krank( B ) being the Kruskal rank of Uand V , respectively.
The Kruskal rank of matrix U is defined as the largest number τ such thatevery set of τ columns of U is linearly independent.
For Q ⊤ ⊙ (cid:98) A , we do not have control over
Q since it is an optimization variable.
But the Kruskal rank k Q ⊤ is at least 1 since we have theconstraint ∥ Q ∥ 0 = MD .
In terms of (cid:98) A , we can show that the matrix (cid:98) A has full Kruskal rankmin (cid:16) (cid:101) K ( (cid:101) K +1) , M (cid:17) .
The detailed proof can be found in Appendix B.
To derive the conditions required for model identification (i.e., rank( Q ⊤ ⊙ (cid:98) A )
= M ), we considerthe following two cases
a) For the case krank ( (cid:98) A ) = (cid:101) K
( (cid:101) K +1) , it requires
M ≥ (cid:101) K ( (cid:101) K + 1) and 1 + (cid:101) K ( (cid:101) K + 1) ≥ M + 1 , which is M = (cid:101) K ( (cid:101) K + 1).
b) For the case krank ( (cid:98) A ) =
M , it is required thatM ≤ (cid:101) K ( (cid:101) K + 1) and 1 + M ≥ M + 1 , which is (cid:101) K ( (cid:101) K + 1)≥ M.
The above implies that if (cid:101) K ( (cid:101) K + ≥ M is satisfied, then the nonlinear functions g m ( · ) ’s can beremoved almost surely.
■
Finite-Sample Identifiability Analysis
The identifiability analysis in Sec. 4 is based on the assumption that (uncountably) infinite samplesare available.
In practice, there are only finite number of samples.
In addition, the proof of Theorem 1assumes that the f m ( · ) ’s are universal function approximators, which is also overly ideal.
In practice,the function learners, e.g., neural networks, may have nontrivial approximation errors for modelingnonlinear functions.
In this section, we provide analyses under more realistic conditions.
To begin with, note that when only N i.i.d.
samples are available, one could rewrite the finite-sampleversion of (15) as follows
[Equation]
where F : R → R is an invertible function class used for modeling the inverse of function g m ( · ) .
Note that in the population case (i.e., N = ∞ ), (11) and (5) have the same solutions.
To characterizethe finite sample performance of (11) under f ( · ) that has limited expressive power, we will use thefollowing definition and assumptions.
Definition 2 (Function Class and Inverse) Assume g ∈ G is an invertible continuous nonlinearfunction.
We define its inverse class G − 1 as a function class that contains all the u ’s satisfyingu ◦ g
( y )
=
αy + β with α ̸ = 0 , ∀ g ∈ G .
Assumption 1 (Realization Gap) Assume that for any u ∈ G − 1 , there exists f ∈ F such that
[Equation]
Assumption 2 (Boundedness) Assume that the 4th-order derivatives of f m ∈ F and g m ∈ G exist.
In addition, | f ( n ) m ( · ) | and | g ( n ) m ( · ) | are bounded for all n ∈ [4] and m ∈
[ M ] .
Any 4th-order derivativeof q ⊤ k h ( As ℓ ) with k =
[ D ] is bounded by C d .
Assume that A has bounded elements.
Assumption 3 (Neural Network Structure) Assume that f m ( · ) is parameterized by a one-hiddenlayer neural network with R neurons and 1 -Lipschitz nonlinear activation function ζ ( · ) :
R → Rwith ζ (0) = 0 .
[Equation]
where ζ ( y ) =
[ ζ ( y 1 ) , . . .
, ζ ( y R )] ⊤ , w i ∈ R R for i = 1 , 2 .
Next, we show the finite-sample identifiability result for the case where the above neural networkclass is used.
To this end, we use the following lemma:
Lemma 1 Assume that F is the function class defined in Assumption 3 and the input x is bounded by| x m | ≤ C x for all m .
Then, the loss function (cid:13)(cid:13)
Q ⊤ f ( x ℓ ) (cid:13)(cid:13) 2 2 has the following Rademacher complexity
[Equation]
[Equation]
The proof can be found in Appendix C. Note that Lemma 1 indicates that the complexity of thefunction class is proportional to the width of the neural network R and inversely proportional to thesample size N .
With Lemma 1, we give the following theorem:.
Theorem 2 (Finite-Sample Identifiability) Under the generative model (3) , assume that Assumptions 1, 2 and 3 hold.
Suppose that x ℓ for ℓ ∈ [ N ] are i.i.d.
samples from X according to a certaindistribution D .
Denote any solution of (11) as (cid:98) f
= [ (cid:98) f 1 , . .
, (cid:98) f M ] ⊤ and (cid:98)
h m = (cid:98) f m ◦ g m for m ∈ [ M ] .
Then, with probability of at least 1 − δ , it holds that
[Equation]
where σ 2min ( Q ⊤ ⊙ (cid:98) A ) denotes the smallest singular value of the matrix Q ⊤ ⊙ (cid:98) A , which is defined inthe linear system (10) .
The detailed proof is relegated to Appendix D.
The proof consists of three steps: 1) Treat the lossfunction as a supervised regression problem and estimate the “generalization error” using the empiricalerror and Rademacher complexity; 2) Approximate the linear system in (10) using the generalizationerror and numerical differentiation; 3) Bound and characterize the second-order derivatives based onthe approximated linear system.
Note that Theorem 2 implies that the second-order derivatives of (cid:98) h m ( · ) ’s approach zero with sufficiently large sample size N .
However, note that the ν term is not determined by N and it onlydecreases to zero when F is a universal function approximator (which could be attained if R is largeenough).
Therefore, there is always a trade-off between the complexity (encoded by key parameterslike depth and width) of the neural network and the effectiveness of model identification given fixedsample size N .
According to Theorem 2, one hopes to employ an expressive enough neural networkfor canceling g m ( · ) , but excessively increasing the expressiveness would hurt the performance undera given N —this is consistent with our experience in many neural network learning problems.
Optimization Design
In this section, we will design an optimization problem to implement the criterion in (11) Specifically,we propose to the following formulation:
[Equation]
Note that the formulation in (15) does not ensure a dense Q .
Nonetheless, if Q is initialized randomlyusing any continuous distribution, iterative optimizers would never return a Q that contains zeros,if the iterative algorithm can be approximated by a continuous transformation.
We use the neuralnetwork structure defined in Assumption 3 to approximate f m ( · ) .
Note that with sufficiently large
R , the function f m ( · ) is a universal approximator [36,37].
In addition, we introduce another neuralnetwork r ( · ) , with the same structure as that of f ( · ) , to make sure that f ( x ℓ ) can be convertedback to x ℓ .
This idea is from autoencoder [38] and it is a sufficient condition to enforce f ( · ) to beinvertible.
The problem is then reformulated as follows
[Equation]
where the hyperparameter λ is used for balancing the energy between the loss and the regularizerterm.
In addition, we define L =
L 1 + λ L 2 .
The problem can be handled by a block coordinate descent (BCD) algorithm as shown in Algorithm 1.
Note that any stochastic optimizer designed for neural network training (e.g., [27]) can be used herefor the f ( · ) and r ( · ) updates.
We use θ f and θ r to denote the parameters of the corresponding neuralnetworks.
Besides, we denote (cid:98) L and (cid:98) L i for i = 1 , 2 as an estimation of L and L i over a random
7.5 5.0 2.5 0.0 2.5 5.0 7.5 input26 o u t p u t f 1 g 1 g15 10 5 0 5 10 15 input26 o u t p u t f 2 g 2 g1.0 0.5 0.0 0.5 1.0 input26 o u t p u t f 3 g 3 g   2 input26 o u t p u t f 4 g 4 g7.5 5.0 2.5 0.0 2.5 5.0 input26 o u t p u t f 5 g 5 g batch B , respectively.
For example, (cid:98) L 1 = 1 |B| (cid:80) ℓ ∈B ∥ Q ⊤ f ( x ℓ ) ∥ 22 , and (cid:98) L 2 ’s is defined in an identicalway.
The term ∇ θ r (cid:98) L 2 is defined in the same way, with the derivative taken w.r.t.
the parameters ofthe r networks.
We refer to the algorithm as post-nonlinear subspace identification .
This is because Theorem 1 indicates that the method identifies range( S ⊤ ) where S =
[ s 1 , . . .
, s N ] for N ≥ K after removing thenonlinear distortions (technically, this needs d m = 0 for all m (cf.
Theorem 1)—which automaticallyholds with our normal minimization objective).
Data: x ℓ for ℓ = 1 , · · · , N Result: f
[Table]
do
Q ← U [: , M − D : M ] where UDV ⊤ =
SVD ( F ) with F =
[ f ( x 1 ) , · · · , f ( x N )] ; while
[Table]
is not [Table] do
Draw a random batch B ;
[Figure]
∇
[Equation]
(cid:98)
Numerical Experiments
In this section, we showcase the effectiveness of the proposed method using numerical validations.
We generate data using the model in (3).
For the neural networks to represent f m ( · ) and r m ( · ) forall m , we use R = 256 with ReLU activations.
Besides, we use Adam [27] optimizer with theinitial learning rate being 2 e − 4 for the network optimization part.
For the hyperparameters, we setλ = 1 e − 4 .
The nonlinear functions g m ( · ) ’s are selected to be variants of e x , sigmoid( x ) and tanh(x)to guarantee invertibility.
Independent Latent Components
In this simulation, we make K = 3 and M = 5 .
Each of thelatent components s ℓ is drawn from uniform distribution U [ − 1 , 1] with N = 10000 , and the mixingmatrix A is drawn from the normal distribution.
Under this setting, it is obvious that Q ∈ R 5 × 2 .
Weplot the composition (cid:98) h m ( · )
=
(cid:98) f m ◦ g m ( · ) in Fig.
The generative functions are in orange while thecomposite functions are plotted in blue.
One can see that all the 5 composite functions are almostaffine, which means we successfully identify and remove g ( · ) .
Prior works on post-nonlinear ICAoftentimes involve computation and sample-size demanding implementations.
For example, in [6],a nonparameteric density estimation step of the learned components needs to be involved in everyiteration in order to measure and promote statistical independence.
In comparison, the proposedapproach is much more straightforward and easier to implement, as it does not rely on statisticalindependence.
Dependent Latent Components
In this case, we consider the latent components drawn from theprobability simplex as in (6).
The mixing matrix A is still drawn from the normal distribution.
Notethat the latent components are dependent on each other in this case.
Consequently the dimensionof the null space is M − K + 1 because there are only (cid:101) K
= K − 1 free variables in s ℓ .
For this
15 input26 o u t p u t f 1 g 1 g15 10 55 input26 o u t p u t f 2 g 2 g0.5 0.0 0.5 1.0 1.5 input26 o u t p u t f 3 g 3 g3 2 1 0 1 2 3 input26 o u t p u t f 4 g 4 g2.5 0.0 2.5 5.0 7.5 input26 o u t p u t f 5 g 5 g
[Table]
simulation, we set sample size to be N = 10000 , with K = 3 and M = 5 .
Accordingly, thedimension of the null space is M − K + 1 = 3 which makes the dimension of Q matrix 5 × 3 .
Theresults are plotted in Fig.
and it is obvious that the nonlinearity is successfully removed.
Readersmight have noticed that in this case the condition in (9) is actually violated since (cid:101) K
( (cid:101) K +1)
= 3 < 5 .
Interestingly, the method still works.
This is likely because in this numerical example Q ⊤ tends tohave full Kruskal rank while in our proof we only used that krank( Q ⊤ )
= 1 .
It turns out that theidentifiability condition could actually be relaxed if Q ⊤ has full Kruskal rank.
We refer interestedreaders to further analyses in Appendix A.
Validating Theorem
In this part, we will demonstrate the impact of R (i.e., the width of theemployed neural network) on the performance of nonlinearity removal, as stated in Theorem 2.
Tobe specific, we measure the subspace distance (see the definition in [23]) between the estimatedrange( (cid:98) S ⊤ ) and the ground truth range( S ⊤ ) , where S =
[ s 1 , · · · , s N ] .
This value is within [0,1] with0 being the best.
The setting is the same as that of the independent case.
The results are averaged over5 random trials and shown in Table 1.
It is obvious that when the function f m ( · ) is not expressiveenough, i.e., when R = 8 , the performance is far from satisfactory.
This is because the term νdominates in the bound (14).
As the number of neurons increases, the subspace distance keepsdecreasing, which indicates that the latent subspace is well identified.
However, if the function isoverly complex ( R = 1024 ), the performance starts to deteriorate since in this case the R becomesdominant on the right hand side of (14).
In addition, one can see that larger N leads to a betterperformance in general, which also validates our claim in (14).
Conclusion
In this work, we revisited the PNL mixture model from a model identification perspective.
Inparticular, we proposed a new framework to identify and remove the unknown nonlinear distortionsin the data generating/acquisition process.
Compared to existing works, the proposed approachdoes not require the latent components to have strong properties or prior knowledge, e.g., statisticalindependence or probability simplex structures.
Consequently, our new result could offer a PNLmodel identification tool to a much wider spectrum of applications, where the latent components
donot exhibit structural constraints used in prior works.
In addition, we provide analysis of the caseswhere only finite samples and non-universal function learners are available—as in practice.
Beyondthat, we design a constrained optimization scheme to implement the proposed learning criterion witha block coordinate descent algorithm.
Our theoretical claims are supported by the experiment results.
There are a number of limitations noticed during the process of developing our subspace-basedlearning method.
First, the PNL model would incur high computational cost when the data featuredimension M is large, since each dimension is modeled with an individual neural network.
Second,there are meaningful applications where the A matrix may not have a nontrivial orthogonal complement (e.g., in PNL mixture based causal direction estimation [10]), yet this current framework doesnot guarantee nonlinearity removal in these cases.
Third, the PNL problem’s learning criterion is anonconvex optimization problem.
The proposed algorithm does not guarantee to solve the formulatedlearning criterion, which creates a gap between the identifiable results (by assuming optimizationoptimality) and the practically attainable results.
Checklist
For
all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A (c ) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]