Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Extraction of latent sources of complex stimuli is critical for making sense of theworld.
While the brain solves this blind source separation (BSS) problem continuously, its algorithms remain unknown.
Previous work on biologically-plausible
BSS algorithms assumed that observed signals are linear mixtures of statisticallyindependent or uncorrelated sources, limiting the domain of applicability of thesealgorithms.
To overcome this limitation, we propose novel biologically-plausibleneural networks for the blind separation of potentially dependent/correlatedsources.
Differing from previous work, we assume some general geometric, notstatistical, conditions on the source vectors allowing separation of potentiallydependent/correlated sources.
Concretely, we assume that the source vectors aresufﬁciently scattered in their domains which can be described by certain polytopes.
Then, we consider recovery of these sources by the Det-Max criterion, whichmaximizes the determinant of the output correlation matrix to enforce a similarspread for the source estimates.
Starting from this normative principle, and using aweighted similarity matching approach that enables arbitrary linear transformationsadaptable by local learning rules, we derive two-layer biologically-plausibleneural network algorithms that can separate mixtures into sources coming froma variety of source domains.
We demonstrate that our algorithms outperform otherbiologically-plausible BSS algorithms on correlated source separation problems.
Introduction
Our brains constantly and effortlessly extract latent causes, or sources, of complex visual, auditoryor olfactory stimuli sensed by sensory organs [1–11].
This extraction is mostly done without anyinstruction, in an unsupervised manner, making the process an instance of the blind source separation(BSS) problem [12, 13].
Indeed, visual and auditory cortical receptive ﬁelds were argued to be theresult of performing BSS on natural images [1, 2] and sounds [4].
The wide-spread use of BSS in thebrain suggests the existence of generic circuit motifs that perform this task [14].
Consequently, theliterature on biologically-plausible neural network algorithms for BSS is growing [15–19]
Because BSS is an underdetermined inverse problem
BSS algorithms make generative on observations most instances the biologically-plausible BSS algorithms complex stimuli are assumed to be linear mixtures of latent sources
This is particularly fruitful and is used to model, for example, natural images [1, 20] and responses of olfactory neurons to complex odorants
However linear mixing by itself is not sufﬁcient for source identiﬁability further assump- tions are needed
Previous work on biologically-plausible algorithms for BSS of linear mixtures independent [ or uncorrelated [
However these real themselves be
[Figure]
In this paper, we address the limitation imposed by independence assumptions and providebiologically-plausible BSS neural networks that can separate potentially correlated sources.
Weachieve this by considering various general geometric identiﬁability conditions on sources insteadof statistical assumptions like independence or uncorrelatedness.
In particular, 1) we make naturalassumptions on the domains of source vectors–like nonnegativity, sparsity, anti-sparsity or boundedness (Figure 1)–and 2) we assume that latent source vectors are sufﬁciently spread in their domain[24, 25].
Because these identiﬁability conditions are not stochastic in nature, our neural networks areable to separate both independent and dependent sources.
We derive our biologically-plausible algorithms from a normative principle.
A common methodfor exploiting our geometric identiﬁability conditions is to disperse latent vector estimates acrosstheir presumed domain by maximizing the determinant of their sample correlation matrix, i.e.,the Det-Max approach [24, 26–29].
Starting from a Det-Max objective function with constraintsthat specify the domain of source vectors, and using mathematical tools introduced for mappingoptimization algorithms to adaptive Hebbian neural networks [18, 30, 31], we derive two-layeredneural networks that can separate potentially correlated sources from their linear mixtures (Figure2).
These networks contain feedforward, recurrent and feedback synaptic connections updated via
Hebbian or anti-Hebbian update rules.
The domain of latent sources determines the structure of theoutput layer of the neural network (Figure 2, Table 1 and Appendix D).
In summary, our main contributions in this article are the following: • We propose a normative framework for generating biologically plausible neural networks that arecapable of separating correlated sources from their mixtures by deriving them from a Det-Max [Figure] objective function subject to source domain constraints
[Figure]
• Our framework can handle inﬁnitely many source types by exploiting their source domain topology.
[Figure]
• We demonstrate the performance of our networks in simulations with synthetic and realistic data.
[Figure]
1.1 Other related work
Several algorithms for separation of linearly mixed and correlated sources have been proposed outsidethe domain of biologically-plausible BSS.
These algorithms make other forms of assumptions onthe latent sources.
Nonnegative matrix factorization (NMF) assumes that the latent vectors arenonnegative [13, 32–34].
Simplex structured matrix factorization (SSMF)
assumes that the latentvectors are members of the unit-simplex [24, 35, 36].
Sparse component analysis (SCA) often assumesthat the latent vectors lie in the unity ` 1 -norm-ball [29, 37–41].
Antisparse bounded componentanalysis (BCA) assumes latent vectors are in the ` 1 -norm-ball [27, 28, 42].
Recently introducedpolytopic matrix factorization (PMF) extends the identiﬁability-enabling domains to inﬁnitely manypolytopes obeying a particular symmetry restriction [25, 43, 44].
The mapping of optimization algorithms to biologically-plausible neural networks have been formalized in the similarity matching framework [30, 31, 45, 46].
Several BSS algorithms were proposedwithin this framework: 1) Nonnegative Similarity Matching (NSM) [16, 47] separates linear mixtures of uncorrelated nonnegative sources, 2)
[19] separates independent sources, and 3) Bounded
Similarity Matching (BSM) separates uncorrelated anti-sparse bounded sources from ` 1 -norm-ball[18].
BSM introduced a weighted inner product-based similarity criterion, referred to as the weightedsimilarity matching (WSM).
Compared to these algorithm, the neural network algorithms we proposein this article 1) cover more general source domains, 2) handle potentially correlated sources, 3)use a two-layer WSM architecture (relative to single layer WSM architecture of BSM, which isnot capable of generating arbitrary linear transformations) and 4) offer a general framework forneural-network-based optimization of the Det-Max criterion.
Problem statement2.1 Sources
We assume that there are n real-valued sources, represented by the vector s 2 P , where P is aparticular subset of R n .
Our algorithms will address a wide range of source domains.
We list someexamples before giving a more general criterion:
• Bounded sparse sources : A natural convex domain choice for sparse sources is the unit ` 1 normball B ` 1 = { s |
k s k
 1 } (Figure 1.(a)).
The use of ` 1 -norm as a convex (non)sparsity measurehas been quite successful with various applications including sparse dictionary learning/componentanalysis [29, 38, 40, 48, 49] and modeling of V1 receptive ﬁelds [2].• Bounded anti-sparse sources : A common domain choice for anti-sparse sources is the unit ` 1norm-ball: B ` 1 = { s |
k s k
 1 } (Figure 1.(b)).
If vectors drawn from B ` 1 are well-spreadinside this set, some samples would contain near-peak magnitude values simultaneously at all theircomponents.
The potential equal spreading of values among the components justiﬁes the term“anti-sparse” [50] or “democratic” [51] component representations.
This choice is well-suited forboth applications in natural images and digital communication constellations [27, 42].• Normalized nonnegative sources : Simplex structured matrix factorization [24, 35, 36] uses the unitsimplex [34, 52]  = { s | s  0 , 1 T s = 1 } (Figure 1.(c)) as the source domain.
Nonnegativityof sources naturally arises in biological context, for example in demixing olfactory mixtures [53].• Nonnegative bounded anti-sparse sources : A non-degenerate polytopic choice of the nonnegativesources can be obtained through the combination of anti-sparseness and nonnegativity constraints.
This corresponds to the intersection of B ` 1 with the nonnegative orthant R n + , represented asB ` 1 , + = B ` 1 \ R n + [25] (Figure 1.(d)).• Nonnegative bounded sparse sources : Another polytopic choice for nonnegative sources canbe obtained through combination of the sparsity and nonnegativity constraints which yields theintersection of B ` 1 with the nonnegative orthant R + , [25]: B ` 1 , + = B ` 1 \ R n + (Figure 1.(e)).
Except the unit simplex  , all the examples above are examples of an inﬁnite set of identiﬁable polytopes whose symmetry groups are restricted to the combinations of component permutations and sign alterations as formalized in PMF framework for BSS [43].
Further, instead of a homogeneous choice of features, such as sparsity and nonnegativity, globally imposed on all elements of the component vector, we can assign these attributes at the subvectorlevel and still obtain identiﬁable polytopes.
For example, the reference [25] provides the set
[-P ex = ⇢ s 2 R s 1 , s 2 
-][Equation]
 [Equation] 1  , as a simple illustration of such polytopes with heterogeneous structure where s 3 is nonnegative, s 1 , s 2 are signed,and [ ss 2 ] T , [ ss 3 ] T are sparse subvectors, while sparsity is not globally imposed.
In thisarticle, we concentrate on particular source domains including the unit simplex, and the subset ofidentiﬁable polytopes for which the attributes such as sparsity and nonnegativity are deﬁned at thesubvector level in the general form
[Equation]
where I + ✓ Z n is the index set for nonnegative sources, and I s is its complement, s J k is the subvector 118 constructed from the elements with indices in
J k , and L is the number of sparsity constraints imposedin the subvector level.
The Det-Max criterion for BSS is based on the assumption that the source samples are well-spread intheir presumed domain.
The references [54] and [25] provide precise conditions on the scattering ofsource samples which guarantee their identiﬁability for the unit simplex and polytopes, respectively.
Appendix A provides a brief summary of these conditions.
We emphasize that our assumptions about the sources are deterministic.
Therefore, our proposedalgorithms do not exploit any stochastic assumptions such as independence or uncorrelatedness, andcan separate both independent and dependent (potentially correlated) sources.
2.2 Mixing
The sources s t are mixed through a mixing matrix A 2 R m ⇥ n .
[Equation]
We only consider the (over)determined case with m  n and assume that the mixing matrix is fullrank.
While we consider noiseless mixtures to achieve perfect separability, the optimization settingproposed for the online algorithm features a particular objective function that safeguards againstpotential noise presence.
We use S ( t ) =
[ s. . .
s t ] 2 R n ⇥ t and X ( t ) =
[ x. . .
x t ]
R m ⇥ t to represent data snapshot matrices, at time t , for sources and mixtures, respectively.
2.3 Separation
The goal of the source separation is to obtain an estimate of S ( t ) from the mixture measurements
X ( t ) when the mixing matrix
A is unknown.
We use the notation y t to refer to source estimates,which are linear transformations of observations, i.e., y i
=
Wx i , where W 2 R n ⇥ m .
We deﬁne
Y ( t ) =
[ yy. . .
y t ] 2 R n ⇥ t as the output snapshot matrix.
"Ideal separation" is deﬁnedas the condition where the outputs are scaled and permuted versions of original sources, i.e., theysatisfy y t =
P ⇤ s t , where P is a permutation matrix, and ⇤ is a full rank diagonal matrix.
Determinant maximization based blind source separation
Among several alternative solution methods for the BSS problem, the determinant-maximization (DetMax) criterion has been proposed within the NMF, BCA, and PMF frameworks, [25–27, 29, 34, 43].
Here, the separator is trained to maximize the (log)-determinant of the sample correlation matrixfor the separator outputs, J ( W ) =
log(det( ˆ R y ( t ))) , where ˆ R y ( t ) is the sample correlation matrixˆ R y ( t ) = 1 t
{+P ti =
 y i+}
y Ti = 1 t Y ( t ) Y ( t ) T .
Further, during the training process, the separator outputsare constrained to lie inside the presumed source domain, i.e. P .
As a result, we can pose thecorresponding optimization problem as [25, 34]
[Equation]
where we ignored the constant 1 t term.
Here, the determinant of the correlation matrix acts as a spreadmeasure for the output samples.
If the original source samples { s 1 , . . .
, s t } are sufﬁciently scatteredinside the source domain P , as described in Section 2.1 and Appendix A, then the global solution ofthis optimization can be shown to achieve perfect separation [25, 34, 54].
An
based on weighted similarity matching
Here, we reformulate the Det-Max problem 3 described above in a way that allows derivation of abiologically-plausible neural network for the linear BSS setup in Section 2. Our formulation appliesto all source types discusses in 2.1.
We propose the following optimization problem:
[Equation]
subject to
[Equation]
Here, X ( t ) 2 R m ⇥ t is the matrix containing input (mixture) vectors, Y ( t ) 2 R n ⇥ t is the matrixcontaining output vectors, H ( t ) 2 R n ⇥ t is a slack variable containing an intermediate signal{ h
i 2 R n , i = 1 , . . .
, t } , corresponding to the hidden layer of the neural network implementationin Section 5, in its columns H ( t ) =
[ h [Equation]
[Equation]
D l, 11 ( t ) , D l, 22 ( t ) , . . .
, D l,nn ( t ) forl = 1 , 2 are nonnegative slack variables to be described below, and D l is the diagonal matrixcontaining weights D l,ii for i = 1 , . . . , n and l = 1 , 2 .
The constraint (4d) ensures that the outputslie in the presumed domain of sources.
This problem is related to the weighted similarity matching (WSM) objective introduced in [18].
Constraints (4b) and (4c) deﬁne two separate WSM conditions.
In particular, the equality constraint in (4b)is a WSM constraint between inputs and the intermediate signal H ( t ) .
This constraint imposes thatthe pairwise weighted correlations of the signal { h i , i = 1 , . .
, t } are the same as correlations amongthe elements of the input signal { x i , i = 1 , . .
. , t } , i.e., x Ti x j = h Ti D 1 ( t ) h j , 8 i, j 2 { 1 , . .
. , t } .
171 D 1 , 11 ( t ) , D 1 , 22 ( t ) , . . .
, D 1 ,nn ( t ) correspond to inner product weights used in these equalities.
Similarly, the equality constraint in (4c) deﬁnes a WSM constraint between the intermediate signal and outputs.
This equality can be written as h Ti h j = y Ti D 2 ( t ) y j , i, j 2 { 1 , . . .
, t } , and
D 2 , 11 ( t ) , D 2 , 22 ( t ) , . .
, D 2 ,nn ( t ) correspond to the inner product weights used in these equalities.
The optimization involves minimizing the logarithm of the determinant of the weighting matrices.
Now we state the relation between our WSM-based objective and the original Det-Max criterion (3).
Theorem 1.
If X ( t ) is full column-rank, then global optimal Y ( t ) solutions of (3) and (4) coincide.
Proof of Theorem 1.
See Appendix B for the proof.
The proof relies on a lemma that states that theoptimization constraints enforce inputs and outputs to be related by an arbitrary linear transformation.
Biologically-plausible neural networks for WSM-based BSS
The optimization problems we considered so far were in an ofﬂine setting, where all inputs areobserved together and all outputs are produced together.
However, biology operates in an onlinefashion, observing an input and producing the corresponding output, before seeing the next input.
Therefore, in this section, we ﬁrst introduce an online version of the batch WSM-problem (4).
Thenwe show that the corresponding gradient descent algorithm leads to a two-layer neural network withbiologically-plausible local update rules.
5.1 Online optimization setting for WSM-based BSS
We ﬁrst propose an online extension of WSM-based BSS (4).
In the online setting, past outputscannot be altered, but past inputs and outputs still carry valuable information about solving the BSS problem.
We will write down an optimization problem whose goal is to produce the sources y t givena mixture x t , while exploiting information from all the ﬁxed previous inputs and outputs.
We ﬁrst introduce our notation.
We consider exponential weighting of the signals as a recipe fordynamical adjustment to potential nonstationarity in the data.
We deﬁne the weighted input datasnapshot matrix by time t as, X ( t ) = ⇥
t  1 x. . .
x t x t ⇤ =
X ( t )  ( t ) , where  is theforgetting factor and  ( t ) =
diag (  t  1 , . .
,  , 1) .
The exponential weighting emphasizes recentmixtures by reducing the impact of past samples.
Similarly, we deﬁne the corresponding weightedoutput snapshot matrix for output as Y ( t ) = ⇥
t  1 y. . .
y t
y t ⇤ =
Y ( t )  ( t ) , and thehidden layer vectors as H ( t ) = ⇥
t  1 h. . .
h t h t ⇤ =
H ( t )  ( t ) .
We further deﬁne⌧ = lim t !1 =
[-P t  1 k =
-][Equation]
[Equation]  2 as a measure of the effective time window length for samplecorrelation calculations based on the exponential weights.
In order to derive an online cost function, we ﬁrst converted equality constraints in (4b) and (4c)to similarity matching cost functions J 1 ( H ( t ) , D 1 ( t ))
= 12 ⌧ 2 k X ( t ) T X ( t )
H ( t ) T D 1 ( t ) H ( t ) k 2 F ,
J 2 ( H ( t ) , D 2 ( t ) , Y ( t ))
= 12 ⌧ 2 k H ( t ) T H ( t )  Y ( t ) T D 2 ( t ) Y ( t )
k 2 F .
Then, a weighted combinationof similarity matching costs and the objective function in (4a) yields the ﬁnal cost function
J ( H ( t ) , D
[Equation]
Here,  2 [0 , 1] and  SM 2 [0 , 1] are parameters that convexly combine similarity matching costsand the objective function.
Finally, we can state the online optimization problem for determining thecurrent output y t , the corresponding hidden state h t and for updating the gain parameters D l ( t ) forl = 1 , 2 , as
[Equation]
subject to
[Equation]
As shown in Appendix C.1, part of J that depends on h t and y t can be written as
[Equation]
where the dependence on past inputs and outputs appear in the weighted correlation matrices:
[Equation]
5.2 Description of network dynamics for bounded anti-sparse sources
We now show that the gradient-descent minimization of the online WSM cost function in (6) canbe interpreted as the dynamics of a neural network with local learning rules.
The exact networkarchitecture is determined by the presumed identiﬁable source domain P , which can be chosen ininﬁnitely many ways.
In this section, we concentrate on the domain choice P = B 1 as an illustrative 217 example.
In Section 5.3, we discuss how to generalize the results of this section by modifying theoutput layer for different identiﬁable source domains.
We start by writing the update expressions forthe optimization variables based on the gradients of J ( h t , y t , D 1 ( t ) , D 2 ( t )) :
Update dynamics for h t : Following previous work [47, 55], and using the gradient of (7) in (A.8)with respect to h t , we can write down an update dynamics for h t in the form
[Equation]
where  H ( t ) is a diagonal matrix containing diagonal elements of M H ( t ) and ¯ M H ( t )
=
M H ( t )  H ( t ) ,  ( · ) is the clipping function, deﬁned as  A ( x ) =
⇢ x  A  x  A, A sign ( x ) otherwise.
This dynamics can be shown to minimize (7) [55].
Here v ( ⌧ ) is an internal variable that could be interpretedas the voltage dynamics of a biological neuron, and is deﬁned based on a linear transformation of h tin (A.9).
Equation (9) deﬁnes v ( ⌧ ) dynamics from the gradient of (7) with respect to h t in (A.8).
Dueto the positive deﬁnite linear map in (A.9), the expression in (A.8) also serves as a descent directionfor v ( ⌧ ) .
Furthermore,  ( · ) function is the projection onto A B 1 , where [  A, A ] is the presumeddynamic range for the components of h t .
We note that there is no explicit constraint set for h t in theonline optimization setting of Section 5.1, and therefore, A can be chosen as large as desired in theactual implementation.
We included the nonlinearity in (10) to model the limited dynamic range ofan actual (biological) neuron.
Update dynamics for output y t : We write the update dynamics for the output y t , based on (A.11) as
[Equation]
[Equation] for i = 1 , .
[Equation]
n
[Equation]
(12) which is derived using the same approach for h t , where we used the descent direction expression in(A.11), and the substitution in (A.12).
Here,  Y ( t ) is a diagonal matrix containing diagonal elementsof M Y ( t ) and ¯ M Y ( t ) =
M Y ( t )   Y ( t ) .
Note that the nonlinear mapping  1 ( · ) is the projectiononto the presumed domain of sources, i.e., P = B 1 , which is elementwise clipping operation.
The state space representations in (9)-(10) and (11)-(12) correspond to a two-layer recurrent neuralnetwork with input x t , hidden layer activation h t , output layer activation y t , W HX
( W THX ) and
W Y H ( W TY H ) are the feedforward (feedback) synaptic weight matrices for the ﬁrst and the secondlayers, respectively, and ¯ M H and ¯ M Y are recurrent synaptic weight matrices for the ﬁrst and thesecond layers, respectively.
The corresponding neural network schematic is provided in Figure 2.(b).
The gain and synaptic weight dynamics below describe the learning mechanism for this network:
Update dynamics for gains D l,ii :
Using the derivative of the cost function with respect to D 1 ,ii in(A.13), we can write the dynamics corresponding to the gain variable D 1 ,ii as
[Equation]
where µ D 1 corresponds to the learning time-constant.
Similarly, for the gain variable D 2 ,ii , thecorresponding coefﬁcient dynamics expression based on (A.14) is given by
[Equation]
where µ D 2 corresponds to the learning time-constant.
The inverses of the inner product weights D l,ii correspond to homeostatic gain parameters.
Theinspection of the gain updates in (13) and (14) leads to an interesting observation: whether thecorresponding gain is going to increase or decrease depends on the balance between the norms of therecurrent and the feedforward synaptic strengths, which are the statistical indicators of the recentoutput and input activations, respectively.
Hence, the homeostatic gain of the neuron will increase(decrease) if the level of recent output activations falls behind (surpasses) the level of recent inputactivations to balance input/output energy levels.
The resulting dynamics align with the experimentalhomeostatic balance observed in biological neurons [56].
Based on the deﬁnitions of the synaptic weight matrices in (8), we can write their updates as
[Equation]
These updates are local in the sense that they only depend on variables available to the synapse, andhence are biologically plausible.
5.3 Det-max WSM neural network
examples for more general source domains
Det-Max Neural Network obtained for the source domain P = B 1 in Section 5.2 can be extended tomore general identiﬁable source domains by only changing the output dynamics.
In Appendix D,we provide illustrative examples for different identiﬁable domain choices.
Table 1 summarizes theoutput dynamics obtained for the identiﬁable source domain examples in Figure
[Table]
[Table] We can make the following observations on Table 1: (1) For sparse and unit simplex settings,there is an additional inhibitory neuron which takes input from all outputs and whose activation isthe inhibitory signal  1 ( ⌧ ) , (2) The source attributes, which are globally deﬁned over all sources,determine the activation functions at the output layer.
The proposed framework can be applied toany polytope described by (1) for which the corresponding Det-Max neural network will containcombinations of activation functions in Table 1 as illustrated in Figure 2.(a).
Numerical experiments
In this section, we illustrate the applications of the proposed WSM-based source separation frameworkfor both synthetic and natural sources.
More details on these experiments and additional examplesare provided in Appendix E, including applications to sparse dictionary learning (Appendix E.6).
6.1 Synthetically correlated source separation
[Figure]
In order to illustrate the correlated sourceseparation capability of the proposed WSMneural networks, we consider a numericalexperiment with ﬁve copula-T distributed(uniform and correlated) sources.
Forthe correlation calibration matrix for thesesources, we use Toeplitz matrix whose ﬁrstrow is [1 ⇢ ⇢ ⇢ ⇢ ] .
The ⇢ parameter determines the correlation level, andwe considered the range [0 , 0 . 6] for this parameter.
These sources are mixed with a10 ⇥ 5 random matrix with independentand identically distributed (i.i.d.) standardnormal random variables.
The mixturesare corrupted by i.i.d.
normal noise corresponding to 30 dB signal-to-noise ratio(SNR) level.
In this experiment, we employ the nonnegative-antisparse-WSM neural network (Figure 8 in Appendix D.2) whose activation functions at the output layer are nonnegative-clipping functions, as the sources are nonnegativeuniform random variables.
We compared the signal-to-interference-plus-noise-power-ratio (SINR) performance of our algorithmwith the NSM algorithm
[16], Infomax ICA algorithm [1], as implemented in Python MNE Toolbox[57], LD-InfoMax algorithm [58], and PMF algorithm [25].
Figure 3 shows the SINR performancesof these algorithms (averaged over 300 realizations) as a function of the correlation parameter ⇢ .
Weobserve that our WSM-based network performs well despite correlations.
In contrast, performance of NSM and ICA algorithms, which assume uncorrelated sources, degrade noticeably with increasingcorrelation levels.
In addition, we note that the performance of batch Det-Max algorithms, i.e.,
LD-InfoMax and PMF, are also robust against source correlations.
Furthermore, due to their batchnature, these algorithms typically achieved better performance results than our neural network withonline-restriction, as expected.
6.2 Image separation
[Figure]
To further illustrate the correlated source separation advantage of our approach, we consider a naturalimage separation scenario.
For this example, we have three RGB images with sizes 324 ⇥ 432 ⇥ 3as sources (Figure 4.(a)).
The sample Pearson correlation coefﬁcients between the images are⇢ 12 = 0 .
263 , ⇢ 13 = 0 .
066 , ⇢ 23 = 0 .
We use a random 5 ⇥ 3 mixing matrix whose entries aredrawn from i.i.d standard normal distribution.
The corresponding mixtures are shown in Figure 4.(b).
[Figure]
We applied ICA, NSM and WSM algorithms to the mixtures.
Figure 4.(c),(d),(e) shows the corresponding outputs.
High-resolution versions of all images in this example are available in Appendix
E.4 in addition to the comparisons with LD-Infomax and PMF algorithms.
The Infomax ICAalgorithm’s outputs have SINR level of 13 . 92 dB, and this performance is perceivable as residualinterference effects in the corresponding output images.
The NSM algorithm achieves signiﬁcantlyhigher SINR level of 17 .
45 dB and the output images visually reﬂect this better performance.
Ouralgorithm achieves the best SINR level of 27 . 49 dB, and the corresponding outputs closely resemblethe original source images.
Discussion and Conclusion
We proposed a general framework for generating biologically plausible neural networks that arecapable of separating correlated sources from their linear mixtures, and demonstrated their successfulcorrelated source separation capability through synthetic and natural sources.
Our networks suffer from the same limitations of other recurrent biologically-plausible BSS networks.
First, certain hyperparameters can signiﬁcantly inﬂuence algorithm performance (see Appendix E.9).
Especially, the inner product gains ( D ii ) are sensitive to the combined choices of algorithm parameters, which require careful tuning.
Second, the numerical experiments with our neural networksare relatively slow due to the recursive computations in (9)-(10) and (11)-(12) for hidden layer andoutput vectors, which is common to all biologically plausible recurrent source separation networks(see Appendix F).
This could perhaps be addressed by early-stopping the recursive computation [59].