Provable Subspace Identiﬁcation Under Post-Nonlinear Mixtures
Abstract
Unsupervised mixture learning (UML) aims at identifying linearly or nonlinearly mixed latent components in a blind manner.
UML is known to be challenging:
Even learning linear mixtures requires highly nontrivial analytical tools, e.g., independent component analysis or nonnegative matrix factorization.
In this work, the post-nonlinear (PNL) mixture model—where unknown element-wise nonlinear functions are imposed onto a linear mixture—is revisited.
The PNL model is widely employed in different ﬁelds ranging from brain signal classiﬁcation, speech separation, remote sensing, to causal discovery.
To identify and remove the unknown nonlinear functions, existing works often assume different properties on the latent components (e.g., statistical independence or probability-simplex structures).
This work shows that under a carefully designed UML criterion, the existence of a nontrivial null space associated with the underlying mixing system sufﬁces to guarantee identiﬁcation/removal of the unknown nonlinearity.
Compared to prior works, our ﬁnding largely relaxes the conditions of attaining PNL identiﬁability, and thus may beneﬁt applications where no strong structural information on the latent components is known.
A ﬁnite-sample analysis is offered to characterize the performance of the proposed approach under realistic settings.
To implement the proposed learning criterion, a block coordinate descent algorithm is proposed.
A series of numerical experiments corroborate our theoretical claims.
Introduction
Unsupervised mixture learning (UML) has been broadly used in machine learning tasks, e.g., for recovering mixed latent components or extracting informative data representations.
Central to understanding UML is the concept of identiﬁability .
The identiﬁability of UML is concerned with whether the latent components can be identiﬁed (up to certain to inconsequential ambiguities) with guarantees—without using any label or training data.
This is a challenging problem due to the unsupervised nature.
To address the identiﬁability challenge, linear mixture learning problems were tackled via exploiting properties of the latent components, e.g., statistical independence [1], nonnegativity [2–4], quasi-stationarity [5,6], and boundedness [7].
This leads to a cohort of blind source separation (BSS) methods.
However, linear mixture models (LMMs) oftentimes fail to perform well on real-world data, due to the unknown nonlinear distortions arising in data generation and acquisition.
In the past two decades, nonlinear mixture models (NMMs) have attracted unprecedentedly more attention [8–14].
Nonetheless, the identiﬁability of the latent components is much harder to establish under NMMs.
It was shown in [15] that the NMMs are in general not identiﬁable, even with strong assumptions, e.g., statistical independence among the latent components.
To establish NMM identifability, a line of work exploited observable auxiliary variables or speciﬁc priors (i.e., extra information about the data or the latent components on top of statistical independence); see, e.g., [13,14,16,17].
Another line of methods took an alternative route via utilizing the structures of the nonlinear distortions.
In particular, the post-nonlinear mixture (PNL) model [8, 9, 18–20] is a reasonable extension of the classical
LMM by assuming unknown nonlinear transformations individually imposed onto the outputs of a underlying linear mixture.
This kind of “structured NMM” is used in a wide range of applications, e.g. hyperspectral imaging [19,21], brain data classiﬁcation [20,22–24], and causal discovery [12,25,26].
In order to identify the generative model/latent components under the PNL model, a key step is to remove the nonlinear distortions in an unsupervised manner.
To this end, the existing methods often explicitly rely on some properties of the latent components, e.g., statistical independence [8,9,18], availability of multiview data [20] and the sum-to-one and nonnegativity (or probability-simplex) structure of the latent components [19,27].
These structures stem from speciﬁc applications, and thus are meaningful—yet not always available.
A natural question is whether the nonlinear distortions are still removable if the widely used structural assumptions (e.g., those in [8,9,18–20,27]) do not hold?
In this work, we offer a positive answer to the above inquiry—and build a new learning algorithm based on our analyses.
Our detailed contributions are as follows.
Subspace-Based PNL Model Identiﬁcation.
We propose a new nonlinearity identiﬁcation and removal criterion under the PNL model.
We show that, as long as the underlying unknown linear mixing system of the PNL model admits a nontrivial orthogonal complement (or, its transpose has a nontrivial null space), the unknown nonlinear transformations are identiﬁable and removable, under mild conditions.
Notably, our approach does not use any speciﬁc structural assumptions of the latent components.
Our ﬁnding presents a fairly different result compared to existing nonlinear mixture learning techniques.
The latter often heavily rely on the properties of the latent components.
Hence, our result may offer new insights and a useful alternative to this long-standing challenge.
An immediate consequence is that our method can work with various types of latent components, including those not covered by existing PNL model identiﬁcation approaches in [8,9,18–20,27].
Performance Analysis Under Realistic Settings.
The model identiﬁcation analysis is conducted under the population case where unlimited data is available, with the assumption that a universal function learner can be used to cancel any nonlinear distortion.
To advance the understanding under more realistic settings, we also consider the realistic scenario where one only has access to ﬁnite samples and non-universal function learners.
The analysis is a combination or statistical generalization theory and numerical differentiation, which is reminiscent of the recently developed framework in [20,28,29]—with nontrivial additional efforts to accommodate our learning criterion.
Optimization Algorithm Design.
Based on the theoretical analysis, we formulate the identiﬁcation problem as a constrained optimization criterion.
The optimization problem is challenging, as it involves learning unknown nonlinear functions [represented by neural networks (NNs)] under nonconvex constraints.
We propose a block coordinate descent (BCD) algorithm that alternates between two subproblems, namely, null space basis learning and nonlinear function learning.
This way, the latter is disentangled with constraints, and thus off-the-shelf NN training techniques (e.g.,
Adam [30]) can be readily employed to handle the subproblem.
A series of experiments are conducted for validating our theoretical claims.
Notations.
x, x , and X represent a scalar, a vector, and a matrix, respectively; k x k 0 , k x k 1 , and k x k 2 denote the ` 0 function, the ` 1 norm, and the Euclidean norm, respectively;  and ~ denote the Khatri-Rao product and the Hadamard product, respectively; f is used to denote a function f ( · ) : R !
R ; f 0 , f 00 and f ( n ) denote the ﬁrst-order, the second-order, and the n th order derivatives of the function f , respectively; f  g denotes the function composition; R ( X ) and N ( X ) mean the range space and the null space of X , respectively.
Background 2.1 UML and Model Identiﬁcation
In UML, the classical LMM is deﬁned as follows [31]:
[Equation]
where x ` 2 R M is the ` th observation sample, s ` 2 R K contains the K latent components, and
A 2 R M ⇥ K is the mixing system.
The latent components in (1) are generally not identiﬁable, if A is not known [31].
As a remedy, information about A and/or s ` is often used to establish identiﬁability.
For instance, statistical independence among [ s ` ] i ’s is used in independent component analysis (ICA) [1], and the nonnegativity of A 2 R M ⇥ K + and S =
[ s 1 , · · · , s N ] 2 R K ⇥ N + are used in nonnegative matrix factorization (NMF)
[3,4]; also see [5–7,32–34] for more conditions that can assist LMM identiﬁcation.
The LMM in (1) is often over-simpliﬁed for modeling the data generation/acquisition processes in real-world applications—where nonlinear effects are widely observed.
In many cases, the NMMs are of more interest.
A typical NMM is as follows [35]:
[Equation]
where g : R K ! R M is a nonlinear “mixing system”.
Compared to the LMM, the NMM in (2) can represent much more complex data generation/acquisition processes.
However, it was shown in [35] that the model in (2) is not identiﬁable, even if the latent components satisfy some fairly strong assumptions, e.g., statistical independence.
A workaround is to use more information of or additional conditions on the latent components, e.g., the availability of auxiliary variables [13,14,16].
Another approach is by exploiting structural assumptions of the nonlinear mixing process.
This leads to the suite of PNL model learning works (see, e.g., [8,9,18–20,27,36]), which will be our focus.
2.2 Post-Nonlinear Mixture Model
The PNL model is a natural extension to the LMM, which is expressed as follows [8,9,18,20,27,36]:
[Equation]
where g =
[ g 1 ( · ) , · · · , g M ( · )] > with each g m ( · ) : R !
R being an invertible nonlinear function.
We assume s ` ⇠ p ( s ) and the support of p ( s ) is a continuous and open domain S .
The PNL model has a myriad of of applications across different ﬁelds, e.g., causal discovery [12,25, 26], hyperspectral imaging [21] and brain data classiﬁcation [22–24].
It is considered as an appropriate model when the unknown nonlinear distortions happen at the sensor end in data acquisition.
It also offers a valuable generalization to the linear mixture generative models in data analytics, e.g., those used in NMF and data clustering [37,38].
An important remark is that the challenge of UML under the PNL model boils down to identifying and removing g ( · ) , as this will lead to a well-understood LMM as in (1).
Then, any LMM identiﬁcation approach, e.g., those in [3,6,7,31,32,34,39], can be applied to identify the latent components s ` for ` = 1 , . . .
, N .
Existing works tackle the g -identiﬁcation and removal problem from various angles.
In [8,18], the statistical independent among the latent components (i.e., [ s ` ] k for k = 1 , . .
, K ) is used.
Recently, [19,27] proposed to identify the PNL model with latent components satisfying the probability simplex constraint (i.e., 1 > s ` = 1 , s `  0 ) that often arises in soft clustering problems.
The work in [20] shows that g ( · ) can be provably removed if multiple views of data exist.
These methods offered insightful and useful approaches to remove g ( · ) , but the use of speciﬁc structural assumptions on the latent components make their applicability relatively narrow.
We propose to identify and remove the nonlinear distortions via exploiting an underlying subspace that often naturally exists under PNL models, instead of resorting to structural assumptions of s ` as in prior works.
Our method starts with the following observation: If the mixing system A is a tall matrix, N ( A > ) is a nontrivial subspace.
This implies that if only the linear mixture part is considered (i.e., if g ( · ) can be removed completely), there exist vectors v such that
[Equation]
where the nonzero v is any vector from N ( A > ) .
In the next section, we will show how this simple equation in (4) allows for constructing a provable g -identiﬁcation and removal criterion.
Proposed Approach
In this section, we propose a learning criterion to identify/remove the nonlinear transformations g ( · ) .
We ﬁrst consider the population case where inﬁnite data points are available; that is, all x ` ⇠ p ( x ) over the continuous support X are available.
Then, we will move forward to consider the ﬁnite-sample case in Sec. 3.2.
Under the inﬁnite-data assumption, our g -identiﬁcation formulation is as follows: ﬁnd Q , f [Equation] [Equation] =
[ f
[Equation] f [Equation] (5a)
[Equation]
where Q 2 R M ⇥ D , D is the dimension of the null space (i.e., N ( A > ) ), in which D  1 and D < M .
The ` 0 function constraint in (5d) means that Q is constrained to be a dense matrix, which will prove important for establishing identiﬁability; this will become clearer in Sec. 3.1.
In the formulation, f m ( · ) is a nonlinear invertible function that we look for.
Ideally, we hope to learn f m ( · ) =
g  1 m ( · ) which cancels the nonlinear transformation.
Then, Q forms a basis of N ( A > ) .
Note that under the orthogonality constraint in (5d), the trivial solution, i.e., Q = 0 , is avoided.
Besides, the invertibility constraint in 5c is used to prevent other degenerate solutions, e.g., f
( x ` ) =
c for all ` with a constant c , from happening.
3.1 Provable Nonlinearity Removal
We will show that solving (5) guarantees that the nonlinear transformation g ( · ) is removed from the
PNL model.
To this end, we ﬁrst introduce the following condition:
Deﬁnition 1 (Locally Free Components) Consider real-valued random vector v =
[ v 1 , . . .
, v d ] >
R d , where v i resides in a continuous and open set V i ✓ R .
Assume that the following holds for all j p ( v j | v  j ) > 0 , with continuous support V j | v  j given any v  j , where v  j denotes the vector with all entries but v j .
Then, the components in v are all locally free components.
Note that the deﬁnition above means that each v j has the “freedom” to change (at least locally) no matter what value the v  j part takes.
As a special case, statistically independent random variables clearly satisfy the deﬁnition.
However, statistical independence is a much stronger condition.
Dependent variables could also be locally free components.
For instance, vectors from the probability simplex, i.e.,
[Equation]
also satisfy Deﬁnition 1—that is, any K  1 of the K components in v are locally free components.
Generally, if v 1 , . .
. , v d are not completely dependent, the locally free condition is not hard to meet.
Simply speaking, with v i and v j being free variables, we could observe any possible value of v j given a ﬁxed ¯ v i .
Using this notion, consider the functional equation q > h ( Av ) = 0 that holds for all v .
Then, we can always observe
[Equation]
with any ﬁxed ¯ v  j , where v j is from the continuous support of p ( v j | ¯ v  j ) > 0 .
Taking derivative of (7) w.r.t.
v j leads to the following:
[Equation]
as all the terms @ ¯ v i / @ v j = 0 for any i 6 = j .
This property will help us show the following main result:
Theorem 1 (Nonlinearity Identiﬁcation and Removal) Under the model in (3) , assume that the criterion (5) is solved.
In addition, assume that A 2 R M ⇥ K is drawn from any joint absolutely continuous distribution, that e K of the K components of s are locally free components (cf.
Def. 1), and that the learned [Equation] f m  g m is twice differentiable for all m 2 [ M ] .
Suppose that
[Equation]
Then, almost surely, any b f ( · ) that is a solution of (5) satisﬁes
[Equation]
8 m 2 [ M ] , where c m 6 = 0 and d m is a constant, at the limit of inﬁnite data.
Theorem 1 indicates that if any solution b f ( · ) of (5) is found, then the composition b h m = b f m  g m is an afﬁne function for all m .
Notably, the theorem does not require all the K latent components to be locally free components, as long as the condition e K ( e K +1) / 2  M is satisﬁed—which means that some components are even allowed to be completely dependent. Intuitively, if e
K is too small then it means that the remaining K  e K components do not provide useful information, while a larger e K implies that there is more diversity among the latent components to assist nonlinearity removal.
Note that the premise here is that the null space N ( A > ) must exist, i.e., M > K should hold as rank( A )
=
min { M, K } with probability one for any A that is drawn from a joint continuous distribution.
Otherwise, one may fail to learn a nonzero Q .
On the other hand, if M is overly large, i.e., M > e K ( e K +1) / 2 , the model is not identiﬁable as there is a lack of information (i.e., the number of locally free components is not large enough) to pin down
all the g i ( · ) ’s with i = 1 , · · · , M .
We show the proof as follows:
Proof : By solving (5), we have the following equation for all s ` 2 S
[Equation]
where h = f  g is the function composition.
For simplicity, we drop the subscript “ ` ” as the equation holds for any s 2 S .
First, we show the existence of solution to (5).
It is obvious that one can let f = g  1 (i.e., f m = g  1 m for all m ) and make the columns of Q to be an orthogonal basis of N ( A > ) .
In addition, such a Q can always be made dense.
This is because A follows a joint continuous distribution, which indicates that the probability of N ( A > ) being orthogonal to any axis is zero.
Next, we show that such a solution is unique up to only afﬁne ambiguities.
By the assumptions, there are e K out of the K components of s that are locally free.
Thus, by taking the second-order (cross) derivatives of Eq.
10 w.r.t.
each of the e K variables, as shown in (8), we have the following:
[Equation]
where a k is the k th column of A .
Without loss of generality, we have assumed that the ﬁrst e K variables of s are locally free components.
Note that if one can show b h 00 m ([ As ] m ) = {+
+} for any m over any s , then it indicates that b h m ( · ) is an afﬁne function for all m over S .
To this end, we show that Q >  B has full column rank, where the sizes of the matrices involved are Q 2 R M ⇥ D and B 2 R f K ( f K +1) ⇥
M .
By Lemma 1 of [40], for matrices U 2 R a ⇥ m and V 2 R b ⇥ m , U  V has full column rank if krank( U ) + krank( V )  m + 1 with krank( U ) and krank( V ) being the Kruskal rank of U and V , respectively.
The Kruskal rank of matrix U is deﬁned as the largest number ⌧ such that every set of ⌧ columns of U is linearly independent.
For Q >  B , we do not have control over
Q since it is an optimization variable.
But the Kruskal rank k Q > is at least 1 since we have the constraint k Q k 0 = MD .
In terms of B , we can show that the matrix B has full Kruskal rank min  e K ( e K +1) / 2 , M  , almost surely.
The detailed explanation can be found in Appendix B.
To derive the conditions required for model identiﬁcation (i.e., rank( Q >  B ) =
M ), we consider the following two cases:
a) For the case of krank ( B ) =
e K ( e K +1) , rank( Q >  B ) =
M with probability one when the following conditions are met: M  e K ( e K + 1) and 1 + e K ( e K + 1)  M + 1 , which holds if M = e K ( e K + 1) .
b) For the case of krank ( B ) =
M , rank( Q >  B ) =
M with probability one when the following conditions are met:
[Equation]
1 , which holds if
[Equation]
The two cases imply that if e
[Equation]
M is satisﬁed, then the nonlinear functions g m ( · ) ’s can be removed almost surely.
⌅
3.2 Finite-Sample Identiﬁability Analysis
The identiﬁability analysis in Sec.
3.1 is based on the assumption that (uncountably) inﬁnite samples are available.
However, one always has to work with ﬁnite samples in practice.
In addition, the proof of Theorem 1 assumed that the f m ( · ) ’s are universal function approximators.
Nonetheless, in practice, the function learners, e.g., neural networks, may have nontrivial approximation errors for modeling nonlinear functions.
In this section, we provide analysis under more realistic conditions.
To begin with, note that when only N i.i.d.
samples are available, one could rewrite the ﬁnite-sample version of (5) as follows
[Equation]
where F : R ! R is an invertible function class used for modeling the inverse of function g m ( · ) .
In the population case (i.e., N = 1 ), (12) and (5) have the same solutions.
To characterize the ﬁnite sample performance of (12) under f ( · ) that has limited expressive power, we will use the following deﬁnition and assumptions:
Deﬁnition 2 (Function Class and Inverse)
The function g 2 G is an invertible continuous nonlin- ear function.
We deﬁne its inverse class G  1 as a function class that contains all the u ’s satisfying u  g ( y ) =
↵ y +  with ↵ 6 = 0 , 8 g 2 G .
Assumption 1 (Realization Gap) For any u
[Equation]  1 , there exists f 2 F such that sup x 2 X | f
( x m )  u ( x m ) |  ⌫ , 8 m 2 [ M ] .
Assumption 2 (Boundedness)
The 4th-order derivatives of f m 2 F and g m 2 G exist.
In addition, | f ( n ) m ( · ) | and | g ( n ) m ( · ) | are bounded for all n 2 [4] and m 2 [ M ] .
Any 4th-order derivative of q > k h ( As ` ) with k =
[ D ] is bounded by C d .
In addition, A has bounded elements.
Assumption 3 (Neural Network Structure)
The function f m ( · ) is parameterized by a one-hiddenlayer neural network with R neurons and 1 -Lipschitz nonlinear activation function ⇣ ( · ) : R !
R with ⇣ (0) =
[Equation]
where ⇣ ( y ) =
[ ⇣ ( y 1 ) , . . .
, ⇣ ( y R )] > , w i 2 R R for i = 1 , 2 .
Next, we show the ﬁnite-sample identiﬁability result for the case where the above neural network class is used.
Our analysis can be readily generalized to cover more complex neural networks such as deep convolutional neural network (CNN) and ResNet.
However, we use this simple neural network as a showcase, since our goal is to reveal insights other than covering complex neural structures.
Using the neural networks from Assumption 3, we have the following lemma:
Lemma 1 Assume that F is the function class deﬁned in Assumption 3 and the input x is bounded by | x m |  C x for all m .
Then, the loss function
[Equation]
[Equation]
2 has the following Rademacher complexity
[Equation]
The proof can be found in Appendix D. Note that Lemma 1 indicates that the complexity of the function class is proportional to the width of the neural network R and inversely proportional to the sample size N .
With Lemma 1, we have the following theorem:
Theorem 2 (Finite-Sample Identiﬁability) Under the generative model (3) , assume that Assumptions 1, 2 and 3 hold.
Suppose that x ` for ` 2 [ N ] are i.i.d.
samples from X according to a certain distribution D .
Denote any solution of (12) as
[Equation]
m 2 [ M ] .
Then, with probability of at least  , the following holds:
[Equation]
where  2min ( Q >  B ) denotes the smallest singular value of the matrix Q >  B , which is deﬁned in the linear system (11) .
The detailed proof is relegated to Appendix D.
The proof consists of three steps: 1) treat the loss function as a supervised regression problem and estimate the “generalization error” using the empirical error and Rademacher complexity; 2) approximate the linear system in (11) using the generalization error and numerical differentiation; and 3) bound and characterize the second-order derivatives based on the approximated linear system.
Theorem 2 implies that the second-order derivatives of b h m ( · ) ’s approach zero with sufﬁciently large sample size N .
However, the ⌫ term is not determined by N and it only decreases to zero when F is a universal function approximator (which could be attained if R is large enough).
Therefore, there is always a trade-off between the complexity (encoded by key parameters like depth and width) of the neural network and the effectiveness of model identiﬁcation given ﬁxed sample size N .
According to Theorem 2, one hopes to employ an expressive enough neural network for canceling g m ( · ) , but excessively increasing the expressiveness would hurt the performance under a given N —this is consistent with our experience in many neural network learning problems.
Optimization Design
In this section, we will design an optimization scheme to implement the criterion in (12) Speciﬁcally, we propose to the following formulation:
[Equation]
Note that the formulation in (16) does not ensure a dense Q .
Nonetheless, if Q is initialized randomly using any continuous distribution, iterative optimizers would never return a Q that contains zeros, if the iterative algorithm can be approximated by a continuous transformation.
We use the neural network structure deﬁned in Assumption 3 to approximate f m ( · ) .
Note that with sufﬁciently large
R , the function f m ( · ) is a universal approximator [42,43].
In addition, to promote invertibility, we introduce another neural network r ( · ) , with the same structure as that of f ( · ) , to make sure that f ( x ` ) can be converted back to x ` .
This idea is from autoencoder [44].
The problem is then recast as follows:
[Equation]
where the hyperparameter   0 is used for balancing the energy between the loss and the regularizer term.
In addition, we deﬁne L =
L 1 +  L 2 .
The problem can be handled by a block coordinate descent (BCD) algorithm as shown in Algorithm 1.
Note that any stochastic optimizer designed for neural network training (e.g., [30]) can be used here for the f ( · ) and r ( · ) updates.
We use ✓ f and ✓ r to denote the parameters of the corresponding neural networks.
Besides, we denote b L and b L i for i = 1 , 2 as estimates of L and L i over a random batch
B , respectively.
For example,
[Equation] and [Equation] L 2 is deﬁned in an identical way.
The term r ✓ r b L 2 denotes the derivative of b L 2 taken w.r.t.
✓ r .
The other derivative terms are deﬁned follwoing the same manner.
We refer to the algorithm as post-nonlinear subspace identiﬁcation .
This is because Theorem 1 indicates that the method identiﬁes range( S > ) where S =
[ s 1 , . . .
, s N ] for N  K after removing the nonlinear distortions (technically, this needs d m = 0 for all m (cf.
Theorem 1)—which automatically holds if our norm minimization term in the objective becomes zero).
Data: x ` for ` = 1 , · · · , N Result: f
Q U [: , M  D : M
r ✓ f b L r ✓ f b L 1 +  r ✓ f b
L 2 ; r ✓ r b
L r ✓ r b
L
[Table]
[Table]
[Table] [Table]
[Table]
Numerical Experiments 5.1 Synthetic Data
We generate data following the model in (3).
For the neural networks representing f m ( · ) and r m ( · ) , we use R = 256 with ReLU activations.
We use the Adam optimizer [30] with the initial learning rate being 2 e  4 for the network optimization part.
For the hyperparameters, we set  = 1 e  4 .
The nonlinear functions g m ( · ) ’s are selected to be variants of e x , sigmoid( x ) and tanh(x) to guarantee invertibility.
The source code can be found online 2 .
Independent Latent Components.
In this simulation, we make K = 3 and M = 5 .
Each element of s ` is drawn independently from the uniform distribution U [  1 , 1] with N = 10000 , and the mixing matrix A is drawn from the normal distribution.
Under this setting, it is obvious that Q 2 R 5 ⇥ 2 .
Fig.
1 shows the composition [Equation]
g m [Equation]
One can see that all the ﬁve compositions are visually afﬁne, which means that the algorithm has successfully identiﬁed and removed g ( · ) .
Prior works on post-nonlinear ICA oftentimes involve computation and sample-size demanding implementations.
For example, in [8], a nonparameteric density estimation step of the learned components needs to be involved in every iteration in order to measure and promote statistical independence.
In comparison, the proposed approach is much more straightforward and easier to implement, as it does not rely on statistical independence.
Dependent Latent Components.
In this case, we consider the latent components drawn from the probability simplex as in (6).
The mixing matrix A is the same as before.
Note that the latent components are dependent.
Consequently the dimension of the null space is M  K + 1 because there are only e
K = K  1 free variables in s ` .
For this simulation, we set N = 10 , 000 , K = 3 , and
M = 5 .
Accordingly, the dimension of N ( A > ) is M  K + 1 = 3 which makes Q a 5 ⇥ 3 matrix.
Fig.
2 shows the results.
It is obvious that the nonlinear distortions are successfully removed.
Readers might have noticed that in this case the condition in (9) is actually violated since e K ( e K +1) / 2 = 3 < 5 .
Interestingly, the method still works.
This is likely because in this numerical example Q > tends to have full Kruskal rank while in our proof we only used that krank( Q > )
= 1 .
It turns out that the identiﬁability condition could actually be much improved if Q > has full Kruskal rank.
We refer interested readers to further discussions in Appendix A.
Validating Theorem 2.
In this part, we demonstrate the impact of R (i.e., the width of the employed neural network) on the performance of nonlinearity removal, as stated in Theorem 2.
To be speciﬁc, we measure the subspace distance (see the deﬁnition in [20]) between the estimated range( b S > ) and the ground truth range( S > ) , where S =
[ s 1 , · · · , s N ] .
This value is within [0,1] with 0 being the best.
The setting is the same as that of the independent case.
Table 1 shows the results, which are averaged over 5 random trials.
One can see that when f m ( · ) is not expressive enough, i.e., when R = 8 , the performance is far from satisfactory.
This is because the term ⌫ dominates in the bound (15).
As the number of neurons increases, the subspace distance keeps decreasing, which indicates that the latent subspace is well identiﬁed.
However, if the function is overly complex ( R = 1024 ), the performance starts to deteriorate since in this case the R becomes dominant on the right hand side of (15).
In addition, one can see that larger N leads to a better performance in general, which also validates our claim in (15).
5.2 Real Data
Dataset.
We use the human face electroencephalogram (EEG) dataset 3 .
The EEG signals were collected when a subject was shown with pictures of real human faces or scrambled faces.
The scrambled face images were generated from the real faces with Fourier transformation by random phase permutations [45]).
At every sample, the EEG signal is a 130 -dimensional vector x ` , which are measured through 130 sensors (channels) all over the subject’s scalp.
The task is to use x ` for ` = 1 , . . .
, N as the training data to learn a classiﬁer.
The classiﬁer aims to determine whether the subject sees a real face image or scrambled ones when a new x ` is collected.
Metric.
We use a number of unsupervised representation learning (URL) method to extract lowdimensional embeddings of x ` ’s. Then, these embeddings are used to train classiﬁers based on support vector machines (SVM) and logistic regression (LR).
We have N = 27 , 692 samples of x ` , which are split as training, validation and test sets with 24794, 1449, and 1449 samples, respectively.
We use the classiﬁcation error to serve as an indirect measure of the performance.
We follow the hypothesis in [45].
There, the model is x ` ⇡ g ( As ` ) with a PNL structure, where s ` are the brain generated “clean” electronic signals, and A and g ( · ) represent the propagation, mixing, and distortions on the sensor end, respectively [23].
Hence, we use our method to remove g ( · ) and then reduce the dimension of As ` to K 0 via PCA.
Our belief is that if the PNL learning method can remove g ( · ) , then the extracted undistorted signals will beneﬁt training linear classiﬁers such as SVM
[Table] [Table]
and LR.
We include PCA (applied onto the raw data), autoencoder, and nonlinear ICA (nICA) based approaches [9,16] as baselines.
Results.
Table 2 shows the classiﬁcation error.
The results are averaged over 5 random trials.
For all methods, the reduced dimension K 0 (for our method, we have D =
M  K 0 ) is selected over { 10 , 30 , 50 , 70 } using the validation set.
We use fully connected networks that have two hidden layers and 256 neurons on each layer.
The same network structure is used for the autoencoder baseline.
The activation functions are all ReLU.
We use Adam [30] optimizer for neural network updates, where the initial learning rate is 1 e  4 .
One can see that our method has the lowest classiﬁcation error, compared to other baselines.
Note that brain signal classiﬁcation problem is quite challenging, and thus using the raw data only outputs an accuracy that is slightly better than random guesses.
Using our method, the average classiﬁcation errors are reduced by 6% and 7% from the those using the raw data for SVM and LR, respectively, which is substantial.
This also shows the usefulness of the PNL model for EEG data.
The PCA is a linear method that does not capture the nonlinearity of the data, and thus the result under PCA is less promising.
The autoencoder can be understood as a nonlinearity removal method, but it does not ensure provable g ( · ) removal.
The nICA variants also exhibit higher error rates relative to the proposed method.
This may be because the ICA frameworks hinge on structural assumptions on the latent components, which may not always be available in practice.
Conclusion
In this work, we revisited the PNL model from a model identiﬁcation perspective.
We proposed a new framework to identify and remove the unknown nonlinear distortions in the data generating/acquisition process.
Compared to existing works, the proposed approach does not require the latent components to satisfy speciﬁc structural assumptions, e.g., statistical independence.
Consequently, our new result offers a PNL model identiﬁcation tool to a much wider spectrum of applications.
In addition, we provided analysis of the cases where only ﬁnite samples and non-universal function learners are available.
We designed a constrained optimization scheme to implement the proposed learning criterion.
Our theoretical claims were supported by experiment results.
There are a number of limitations.
First, the PNL model would incur high computational cost when the data feature dimension M is large, since each dimension induces an individual neural network.
Second, there are meaningful applications where the A matrix may not have a nontrivial orthogonal complement (e.g., in PNL-based causal direction estimation [12]), yet this current framework does not guarantee nonlinearity removal in those cases.
Third, the PNL problem’s learning criterion is a nonconvex optimization problem.
The proposed algorithm does not guarantee to solve the formulated learning criterion, which creates a gap between the identiﬁable results (by assuming optimization optimality) and the attainable results in practice.
Acknowledgement : This work was supported by the National Science Foundation (NSF) CAREER
Award under Project ECCS-2144889.