Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Abstract
Extraction of latent sources of complex stimuli is critical for making sense of the world.
While the brain solves this blind source separation (BSS) problem continuously, its algorithms remain unknown.
Previous work on biologically-plausible
BSS algorithms assumed that observed signals are linear mixtures of statistically independent or uncorrelated sources, limiting the domain of applicability of these algorithms.
To overcome this limitation, we propose novel biologically-plausible neural networks for the blind separation of potentially dependent/correlated sources.
Differing from previous work, we assume some general geometric, not statistical, conditions on the source vectors allowing separation of potentially dependent/correlated sources.
Concretely, we assume that the source vectors are sufﬁciently scattered in their domains which can be described by certain polytopes.
Then, we consider recovery of these sources by the Det-Max criterion, which maximizes the determinant of the output correlation matrix to enforce a similar spread for the source estimates.
Starting from this normative principle, and using a weighted similarity matching approach that enables arbitrary linear transformations adaptable by local learning rules, we derive two-layer biologically-plausible neural network algorithms that can separate mixtures into sources coming from a variety of source domains.
We demonstrate that our algorithms outperform other biologically-plausible BSS algorithms on correlated source separation problems.
Introduction
Our brains constantly and effortlessly extract latent causes, or sources, of complex visual, auditory or olfactory stimuli sensed by sensory organs [1–11].
This extraction is mostly done without any instruction, in an unsupervised manner, making the process an instance of the blind source separation (BSS) problem [12, 13].
Indeed, visual and auditory cortical receptive ﬁelds were argued to be the result of performing BSS on natural images [1, 2] and sounds [4].
The wide-spread use of BSS in the brain suggests the existence of generic circuit motifs that perform this task [14].
Consequently, the literature on biologically-plausible neural network algorithms for BSS is growing [15–19].
Because BSS is an underdetermined inverse problem, BSS algorithms make generative assumptions on observations.
In most instances of the biologically-plausible BSS algorithms, complex stimuli are assumed to be linear mixtures of latent sources.
This assumption is particularly fruitful and is used to model, for example, natural images [1, 20], and responses of olfactory neurons to complex odorants [21–23].
However, linear mixing by itself is not sufﬁcient for source identiﬁability; further assumptions are needed.
Previous work on biologically-plausible algorithms for BSS of linear mixtures 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[Figure]
assumed sources to be statistically independent [17, 19, 24] or uncorrelated [16, 18].
However, these assumptions are very limiting when considering real data where sources can themselves be correlated.
In this paper, we address the limitation imposed by independence assumptions and provide biologically-plausible BSS neural networks that can separate potentially correlated sources.
We achieve this by considering various general geometric identiﬁability conditions on sources instead of statistical assumptions like independence or uncorrelatedness.
In particular, 1) we make natural assumptions on the domains of source vectors–like nonnegativity, sparsity, anti-sparsity or boundedness (Figure 1)–and 2) we assume that latent source vectors are sufﬁciently spread in their domain [25, 26].
Because these identiﬁability conditions are not stochastic in nature, our neural networks are able to separate both independent and dependent sources.
We derive our biologically-plausible algorithms from a normative principle.
A common method for exploiting our geometric identiﬁability conditions is to disperse latent vector estimates across their presumed domain by maximizing the determinant of their sample correlation matrix, i.e., the Det-Max approach [25, 27–30].
Starting from a Det-Max objective function with constraints that specify the domain of source vectors, and using mathematical tools introduced for mapping optimization algorithms to adaptive Hebbian neural networks [18, 31, 32], we derive two-layered neural networks that can separate potentially correlated sources from their linear mixtures (Figure 2).
These networks contain feedforward, recurrent and feedback synaptic connections updated via
Hebbian or anti-Hebbian update rules.
The domain of latent sources determines the structure of the output layer of the neural network (Figure 2, Table 1 and Appendix D).
In summary, our main contributions in this article are the following
: • We propose a normative framework for generating biologically plausible neural networks that are capable of separating correlated sources from their mixtures by deriving them from a Det-Max objective function subject to source domain constraints.
• Our framework can handle inﬁnitely many source types by exploiting their source domain topology.
• We demonstrate the performance of our networks in simulations with synthetic and realistic data.
1.1 Other related work
Several algorithms for separation of linearly mixed and correlated sources have been proposed outside the domain of biologically-plausible BSS.
These algorithms make other forms of assumptions on the latent sources.
Nonnegative matrix factorization (NMF) assumes that the latent vectors are nonnegative [13, 33–35].
Simplex structured matrix factorization (SSMF)
assumes that the latent vectors are members of the unit-simplex [25, 36, 37].
Sparse component analysis (SCA) often assumes that the latent vectors lie in the unity ` 1 -norm-ball [30, 38–42].
Antisparse bounded component analysis (BCA) assumes latent vectors are in the ` 1 -norm-ball [28, 29, 43].
Recently introduced polytopic matrix factorization (PMF) extends the identiﬁability-enabling domains to inﬁnitely many polytopes obeying a particular symmetry restriction [26, 44, 45].
The mapping of optimization algorithms to biologically-plausible neural networks have been formalized in the similarity matching framework [31, 32, 46, 47].
Several BSS algorithms were proposed within this framework: 1) Nonnegative Similarity Matching (NSM)
[16, 48] separates linear mixtures of uncorrelated nonnegative sources, 2)
[19] separates independent sources, and 3) Bounded
Similarity Matching (BSM) separates uncorrelated anti-sparse bounded sources from ` 1 -norm-ball
[Figure]
BSM introduced a weighted inner product-based similarity criterion, referred to as the weighted similarity matching (WSM).
Compared to these algorithm, the neural network algorithms we propose in this article 1) cover more general source domains, 2) handle potentially correlated sources, 3) use a two-layer WSM architecture (relative to single layer WSM architecture of BSM, which is not capable of generating arbitrary linear transformations) and 4) offer a general framework for neural-network-based optimization of the Det-Max criterion.
Problem statement 2.1 Sources
We assume that there are n real-valued sources, represented by the vector s 2 P , where P is a particular subset of R n .
Our algorithms will address a wide range of source domains.
We list some examples before giving a more general criterion:
• Bounded sparse sources : A natural convex domain choice for sparse sources is the unit ` 1 norm ball B ` 1 = { s |
{+k s k 
  1 } (Figure 1.(a)).+} The use of ` 1 -norm as a convex (non)sparsity measure has been quite successful with various applications including sparse dictionary learning/component analysis [30, 39, 41, 49, 50] and modeling of V1 receptive ﬁelds [2].
• Bounded anti-sparse sources : A common domain choice for anti-sparse sources is the unit ` 1norm-ball: B ` 1 = { s |
k s {+k 
+}
 1 } (Figure 1.(b)).
If vectors drawn from B ` 1 are well-spread inside this set, some samples would contain near-peak magnitude values simultaneously at all their components.
The potential equal spreading of values among the components justiﬁes the term “anti-sparse” [51] or “democratic” [52] component representations.
This choice is well-suited for both applications in natural images and digital communication constellations [28, 43].
• Normalized nonnegative sources : Simplex structured matrix factorization [25, 36, 37] uses the unit simplex [35, 53]  = { s | s  0 , 1 T s = 1 } (Figure 1.(c)) as the source domain.
Nonnegativity of sources naturally arises in biological context, for example in demixing olfactory mixtures [54].
• Nonnegative bounded anti-sparse sources : A non-degenerate polytopic choice of the nonnegative sources can be obtained through the combination of anti-sparseness and nonnegativity constraints.
This corresponds to the intersection of B ` 1 with the nonnegative orthant R n + , represented as B ` 1 , + = B ` 1 \ R n + [26] (Figure 1.(d)).
• Nonnegative bounded sparse sources : Another polytopic choice for nonnegative sources can be obtained through combination of the sparsity and nonnegativity constraints which yields the intersection of B ` 1 with the nonnegative orthant R + , [26]: B ` 1 , + = B ` 1 \ R n + (Figure 1.(e)).
Except the unit simplex  all the examples above are examples of an inﬁnite set of identiﬁable polytopes whose symmetry groups are restricted to the combinations of component permutations and sign alterations as formalized in PMF framework for BSS [44].
Further, in- stead of a homogeneous choice of features, such as sparsity and nonnegativity, globally imposed on all elements of the component vector, we can assign these attributes at the subvector level and still obtain identiﬁable polytopes.
For example, the reference [26] provides the set
P
[Equation]

[Equation]
, as a simple illustration of such polytopes with heterogeneous structure where s 3 is nonnegative, s 1 , s 2 are signed, and [ ss 2 ] T , [ ss 3 ] T are sparse subvectors, while sparsity is not globally imposed.
In this article, we concentrate on particular source domains including the unit simplex, and the subset of identiﬁable polytopes for which the attributes such as sparsity and nonnegativity are deﬁned at the subvector level in the general form
[Equation]
where I + ✓ Z n is the index set for nonnegative sources, and I s is its complement, s J k is the subvector constructed from the elements with indices in J k , and L is the number of sparsity constraints imposed in the subvector level.
The Det-Max criterion for BSS is based on the assumption that the source samples are well-spread in their presumed domain.
The references [55] and [26] provide precise conditions on the scattering of source samples which guarantee their identiﬁability for the unit simplex and polytopes, respectively.
Appendix A provides a brief summary of these conditions.
We emphasize that our assumptions about the sources are deterministic.
Therefore, our proposed algorithms do not exploit any stochastic assumptions such as independence or uncorrelatedness, and can separate both independent and dependent (potentially correlated) sources.
2.2 Mixing
The sources s t are mixed through a mixing matrix A 2 R m ⇥ n .
[Equation]
We only consider the (over)determined case with m  n and assume that the mixing matrix is fullrank.
While we consider noiseless mixtures to achieve perfect separability, the optimization setting proposed for the online algorithm features a particular objective function that safeguards against potential noise presence.
We use S ( t ) =
[ s. . .
s t ] 2 R n ⇥ t and X ( t ) =
[ x. . .
x t ] R m ⇥ t to represent data snapshot matrices, at time t , for sources and mixtures, respectively.
2.3 Separation
The goal of the source separation is to obtain an estimate of S ( t ) from the mixture measurements
X ( t ) when the mixing matrix
A is unknown.
We use the notation y t to refer to source estimates, which are linear transformations of observations, i.e., y i
=
Wx i , where W 2 R n ⇥ m .
We deﬁne
Y ( t ) =
[ yy. . .
y t ] 2 R n ⇥ t as the output snapshot matrix.
"Ideal separation" is deﬁned as the condition where the outputs are scaled and permuted versions of original sources, i.e., they satisfy y t = P ⇤ s t , where P is a permutation matrix, and ⇤ is a full rank diagonal matrix.
Determinant maximization based blind source separation
Among several alternative solution methods for the BSS problem, the determinant-maximization (DetMax) criterion has been proposed within the NMF, BCA, and PMF frameworks, [26–28, 30, 35, 44].
Here, the separator is trained to maximize the (log)-determinant of the sample correlation matrix for the separator outputs, J ( W ) =
log(det( ˆ R y ( t ))) , where ˆ R y ( t ) is the sample correlation matrix
ˆ R y ( t ) = 1 t P ti {+=
+}
y i
y Ti = 1 t Y ( t ) Y ( t ) T .
Further, during the training process, the separator outputs are constrained to lie inside the presumed source domain, i.e. P .
As a result, we can pose the corresponding optimization problem as [26, 35]
[Equation]
where we ignored the constant 1 t term.
Here, the determinant of the correlation matrix acts as a spread measure for the output samples.
If the original source samples { s 1 , . . .
, s t } are sufﬁciently scattered inside the source domain P , as described in Section 2.1 and Appendix A, then the global solution of this optimization can be shown to achieve perfect separation [26, 35, 55].
An alternative optimization formulation of determinant-maximization based on weighted similarity matching
Here, we reformulate the Det-Max problem 3 described above in a way that allows derivation of a biologically-plausible neural network for the linear BSS setup in Section 2.
Our formulation applies to all source types discusses in 2.1.
We propose the following optimization problem:
[Equation]
subject to
[Equation]
Here, X ( t ) 2 R m ⇥ t is the matrix containing input (mixture) vectors, Y ( t ) 2 R n ⇥ t is the matrix containing output vectors, H ( t ) 2 R n ⇥ t is a slack variable containing an intermediate signal { h i 2 R n , i = 1 , . . .
, t } , corresponding to the hidden layer of the neural network implementation in Section 5, in its columns H ( t ) =
[ hh. . .
h t ] .
D l, 11 ( t ) , D l, 22 ( t ) , . . .
, D l,nn ( t ) for l = 1 , 2 are nonnegative slack variables to be described below, and D l is the diagonal matrix containing weights D l,ii for i = 1 , . . . , n and l = 1 , 2 .
The constraint (4d) ensures that the outputs lie in the presumed domain of sources.
This problem is related to the weighted similarity matching (WSM) objective introduced in [18].
Constraints (4b) and (4c) deﬁne two separate WSM conditions.
In particular, the equality constraint in (4b) is a WSM constraint between inputs and the intermediate signal H ( t ) .
This constraint imposes that the pairwise weighted correlations of the signal { h i , i = 1 , . . .
, t } are the same as correlations among the elements of the input signal { x i , i = 1 , . .
. , t } , i.e., x Ti x j = h Ti D 1 ( t ) h j , 8 i, j 2 { 1 , . .
. , t } .
D 1 , 11 ( t ) , D 1 , 22 ( t ) , . . .
, D 1 ,nn ( t ) correspond to inner product weights used in these equalities.
Similarly, the equality constraint in (4c) deﬁnes a WSM constraint between the intermediate signal and outputs.
This equality can be written as h Ti h j = y Ti D 2 ( t ) y j , i, j 2 { 1 , . . .
, t } , and
D 2 , 11 ( t ) , D 2 , 22 ( t ) , . .
, D 2 ,nn ( t ) correspond to the inner product weights used in these equalities.
The optimization involves minimizing the logarithm of the determinant of the weighting matrices.
Now we state the relation between our WSM-based objective and the original Det-Max criterion (3).
Theorem 1.
If X ( t ) is full column-rank, then global optimal Y ( t ) solutions of (3) and (4) coincide.
Proof of Theorem 1.
See Appendix B for the proof.
The proof relies on a lemma that states that the optimization constraints enforce inputs and outputs to be related by an arbitrary linear transformation.
Biologically-plausible neural networks for WSM-based BSS
The optimization problems we considered so far were in an ofﬂine setting, where all inputs are observed together and all outputs are produced together.
However, biology operates in an online fashion, observing an input and producing the corresponding output, before seeing the next input.
Therefore, in this section, we ﬁrst introduce an online version of the batch WSM-problem (4).
Then we show that the corresponding gradient descent algorithm leads to a two-layer neural network with biologically-plausible local update rules.
5.1 Online optimization setting for WSM-based BSS
We ﬁrst propose an online extension of WSM-based BSS (4).
In the online setting, past outputs cannot be altered, but past inputs and outputs still carry valuable information about solving the BSS problem.
We will write down an optimization problem whose goal is to produce the sources y t given a mixture x t , while exploiting information from all the ﬁxed previous inputs and outputs.
We ﬁrst introduce our notation.
We consider exponential weighting of the signals as a recipe for dynamical adjustment to potential nonstationarity in the data.
We deﬁne the weighted input data snapshot matrix by time t as, X ( t ) = ⇥
t  1 x. . .
x t x t ⇤ = X ( t )  ( t ) , where  is the forgetting factor and  ( t ) =
diag (  t  1 , . .
,  , 1) .
The exponential weighting emphasizes recent mixtures by reducing the impact of past samples.
Similarly, we deﬁne the corresponding weighted output snapshot matrix for output as Y ( t ) = ⇥
t  1 y. . .  y t
y t ⇤ = Y ( t )  ( t ) , and the hidden layer vectors as H ( t ) = ⇥
t  1 h. . .  h t h t ⇤ =
H ( t )  ( t ) .
We further deﬁne ⌧ = lim t !1 =
P t [Equation] = [-(cid:80) t − 1 k =
-][Equation]
= [Equation] [Equation] 2 as a measure of the effective time window length for sample correlation calculations based on the exponential weights.
In order to derive an online cost function, we ﬁrst converted equality constraints in (4b) and (4c) to similarity matching cost functions J 1 ( H ( t ) , D 1 ( t ))
= {+
 [Equation]+} 2 k X ( t ) T X ( t )  H ( t ) T D 1 ( t ) H ( t ) k 2 F ,
J 2 ( H ( t ) , D 2 ( t ) , Y ( t )) =
[Equation] ⌧ 2 k H ( t ) T H ( t )  Y ( t ) T D 2 ( t ) Y ( t )
k 2 F .
Then, a weighted combination of similarity matching costs and the objective function in (4a) yields the ﬁnal cost function
J ( H ( [Equation] , D [Equation]
[Equation]
Here,  2 [0 , 1] and  SM 2 [0 , 1] are parameters that convexly combine similarity matching costs and the objective function.
Finally, we can state the online optimization problem for determining the current output y t , the corresponding hidden state h t and for updating the gain parameters D l ( t ) for l = 1 , 2 , as minimize
[Equation]
subject to
[Equation]
As shown in Appendix C.1, part of J that depends on h t and y t can be written as
[Equation]
where the dependence on past inputs and outputs appear in the weighted correlation matrices:
[Equation]
5.2 Description of network dynamics for bounded anti-sparse sources
We now show that the gradient-descent minimization of the online WSM cost function in (6) can be interpreted as the dynamics of a neural network with local learning rules.
The exact network architecture is determined by the presumed identiﬁable source domain P , which can be chosen in inﬁnitely many ways.
In this section, we concentrate on the domain choice P = B 1 as an illustrative example.
In Section 5.3, we discuss how to generalize the results of this section by modifying the output layer for different identiﬁable source domains.
We start by writing the update expressions for the optimization variables based on the gradients of J ( h t , y t , D 1 ( t ) , D 2 ( t )) :
Update dynamics for h t : Following previous work [48, 56], and using the gradient of (7) in (A.8) with respect to h t , we can write down an update dynamics for h t in the form
[Equation]
where  H ( t ) is a diagonal matrix containing diagonal elements of M H ( t ) and ¯ M H ( t )
=
M H ( t )   H ( t ) ,  ( · ) is the clipping function, deﬁned as  A ( x ) =
⇢ x  A  x  A, A sign ( x ) otherwise.
This dynamics can be shown to minimize (7) [56].
Here v ( ⌧ ) is an internal variable that could be interpreted as the voltage dynamics of a biological neuron, and is deﬁned based on a linear transformation of h t in (A.9).
Equation (9) deﬁnes v ( ⌧ ) dynamics from the gradient of (7) with respect to h t in (A.8).
Due to the positive deﬁnite linear map in (A.9), the expression in (A.8) also serves as a descent direction for v ( ⌧ ) .
Furthermore,  ( · ) function is the projection onto A B 1 , where [  A, A ] is the presumed dynamic range for the components of h t .
We note that there is no explicit constraint set for h t in the online optimization setting of Section 5.1, and therefore, A can be chosen as large as desired in the actual implementation.
We included the nonlinearity in (10) to model the limited dynamic range of an actual (biological) neuron.
Update dynamics for output y t : We write the update dynamics for the output y t , based on (A.11) as
[Equation]
for i = 1 , . . . n
[Equation]
12) which is derived using the same approach for h t , where we used the descent direction expression in (A.11), and the substitution in (A.12).
Here,  Y ( t ) is a diagonal matrix containing diagonal elements of M Y ( t ) and ¯ M Y ( t ) =
M Y ( t )   Y ( t ) .
Note that the nonlinear mapping  1 ( · ) is the projection onto the presumed domain of sources, i.e., P = B 1 , which is elementwise clipping operation.
The state space representations in (9)-(10) and (11)-(12) correspond to a two-layer recurrent neural network with input x t , hidden layer activation h t , output layer activation y t , W HX
( W THX ) and
W Y H ( W TY H ) are the feedforward (feedback) synaptic weight matrices for the ﬁrst and the second layers, respectively, and ¯ M H and ¯ M Y are recurrent synaptic weight matrices for the ﬁrst and the second layers, respectively.
The corresponding neural network schematic is provided in Figure 2.(b).
The gain and synaptic weight dynamics below describe the learning mechanism for this network:
Update dynamics for gains D l,ii :
Using the derivative of the cost function with respect to D 1 ,ii in (A.13), we can write the dynamics corresponding to the gain variable D 1 ,ii as
[Equation]
where µ D 1 corresponds to the learning time-constant.
Similarly, for the gain variable D 2 ,ii , the corresponding coefﬁcient dynamics expression based on (A.14) is given by
[Equation]
where µ D 2 corresponds to the learning time-constant.
The inverses of the inner product weights D l,ii correspond to homeostatic gain parameters.
The inspection of the gain updates in (13) and (14) leads to an interesting observation: whether the corresponding gain is going to increase or decrease depends on the balance between the norms of the recurrent and the feedforward synaptic strengths, which are the statistical indicators of the recent output and input activations, respectively.
Hence, the homeostatic gain of the neuron will increase (decrease) if the level of recent output activations falls behind (surpasses) the level of recent input activations to balance input/output energy levels.
The resulting dynamics align with the experimental homeostatic balance observed in biological neurons [57].
Based on the deﬁnitions of the synaptic weight matrices in (8), we can write their updates as
[Equation]
These updates are local in the sense that they only depend on variables available to the synapse, and hence are biologically plausible.
5.3 Det-max WSM neural network examples for more general source domains
Det-Max Neural Network obtained for the source domain P = B 1 in Section 5.2 can be extended to more general identiﬁable source domains by only changing the output dynamics.
In Appendix D, we provide illustrative examples for different identiﬁable domain choices.
Table 1 summarizes the output dynamics obtained for the identiﬁable source domain examples in Figure 1.
[Table]
P
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
⇣ [Equation] [Table]
[Equation] [Table] Sparse da
[Equation]
[Equation] [Table] [Table] da
[Table]
P [Table] [Table]
, Sparse [Table]
P
[Table]
[Table] [Table]
P
[Table]
We can make the following observations on Table 1: (1) For sparse and unit simplex settings, there is an additional inhibitory neuron which takes input from all outputs and whose activation is the inhibitory signal  1 ( ⌧ ) , (2) The source attributes, which are globally deﬁned over all sources, determine the activation functions at the output layer.
The proposed framework can be applied to any polytope described by (1) for which the corresponding Det-Max neural network will contain combinations of activation functions in Table 1 as illustrated in Figure 2.(a).
Numerical experiments
In this section, we illustrate the applications of the proposed WSM-based BSS framework for both synthetic and natural sources.
More details on these experiments and additional examples are provided in Appendix E, including sparse dictionary learning.
Our implementation code is publicly available 1 .
6.1 Synthetically correlated source separation
In order to illustrate the correlated source separation capability of the proposed WSM neural networks, we consider a numerical experiment with ﬁve copula-T distributed (uniform and correlated) sources.
For the correlation calibration matrix for these sources, we use Toeplitz matrix whose ﬁrst row is [1 ⇢ ⇢ ⇢ ⇢ ] .
The ⇢ parameter determines the correlation level, and we considered the range [0 , 0 . 6] for this parameter.
These sources are mixed with a 10 ⇥ 5 random matrix with independent and identically distributed (i.i.d.)
standard normal random variables.
The mixtures are corrupted by i.i.d.
normal noise corresponding to 30 dB signal-to-noise ratio (SNR) level.
In this experiment, we employ the nonnegative-antisparse-WSM neural network (Figure 8 in Appendix D.2) whose activation functions at the output layer are nonnegative-clipping functions, as the sources are nonnegative uniform random variables.
[Figure]
We compared the signal-to-interferenceplus-noise-power-ratio (SINR) performance of our algorithm with the NSM algorithm [16], Infomax ICA algorithm [1], as implemented in Python MNE
Toolbox [58], LD-InfoMax algorithm [59], and PMF algorithm [26].
Figureshows the SINR performances of these algorithms (averaged over 300 realizations) as a function of the correlation parameter ⇢ .
We observe that our WSM-based network performs well despite correlations.
In contrast, performance of NSM and ICA algorithms, which assume uncorrelated sources, degrade noticeably with increasing correlation levels.
In addition, we note that the performance of batch Det-Max algorithms, i.e., LD-InfoMax and PMF, are also robust against source correlations.
Furthermore, due to their batch nature, these algorithms typically achieved better performance results than our neural network with online-restriction, as expected.
6.2 Image separation
To further illustrate the correlated source separation advantage of our approach, we consider a natural image separation scenario.
For this example, we have three RGB images with sizes 324 ⇥ 432 ⇥as sources (Figure 4.(a)).
The sample Pearson correlation coefﬁcients between the images are ⇢ 12 = 0 .
263 , ⇢ 13 = 0 .
066 , [-ρ 
-]⇢ 23 = 0 .
We use a random 5 ⇥ 3 mixing matrix whose entries are drawn from i.i.d standard normal distribution.
The corresponding mixtures are shown in Figure 4.(b).
[Figure]
We applied ICA, NSM and WSM algorithms to the mixtures.
Figure 4.(c),(d),(e) shows the corresponding outputs.
High-resolution versions of all images in this example are available in Appendix
E.4 in addition to the comparisons with LD-Infomax and PMF algorithms.
The Infomax ICA algorithm’s outputs have SINR level of 13 . 92 dB, and this performance is perceivable as residual interference effects in the corresponding output images.
The NSM algorithm achieves signiﬁcantly higher SINR level of 17 .
45 dB and the output images visually reﬂect this better performance.
Our algorithm achieves the best SINR level of 27 . 49 dB, and the corresponding outputs closely resemble the original source images.
Discussion and Conclusion
We proposed a general framework for generating biologically plausible neural networks that are capable of separating correlated sources from their linear mixtures, and demonstrated their successful correlated source separation capability through synthetic and natural sources.
Another motivation for our work is to link network structure with function.
This is a long standing goal of neuroscience, however examples where this link can be achieved are limited.
Our work provides concrete examples where clear links between a network’s architecture–i.e. number of interneurons, connections between interneurons and output neurons, nonlinearities (frequency-current curves)– and its function, the type of source separation or feature extraction problem the networks solves, can be established.
These links may provide insights and interpretations that might generalize to real biological circuits.
Our networks suffer from the same limitations of other recurrent biologically-plausible BSS networks.
First, certain hyperparameters can signiﬁcantly inﬂuence algorithm performance (see Appendix
E.9).
[-+ ∥ h t ∥ 4 D 1 ( t ) − 2 ∥ x t ∥ 
-]
Especially, the inner product gains ( D ii ) are sensitive to the combined choices of algorithm parameters, which require careful tuning.
[-(cid:13) (cid:13)(cid:13)(cid:13) 1 ≤ 1 , (cid:13) (cid:13)(cid:13)(cid:13)(cid:20) ss(cid:21) (cid:13) (cid:13)(cid:13)(cid:13) 
-]
[-• D 
-]
[-I , and D 
-]
[-• M H = 
-]
[-• D 
-]
[-I , and D 
-]
[-• M H = 
-]
[-• D 
-]
[-I , and D 
-]
[-• M H = 
-]
[-Recall that our WSM-based network outputs illustrated in Figure 14e achieves SINR level of 27.49 dB. LD-InfoMax algorihtm’s outputs in Figure 14f obtain SINR level of 28.65 dB, and the PMF algorithm’s outputs in Figure 
-]
[-• D 1 = 8 I , and D 
-]
Second, the numerical experiments with our neural networks are relatively slow due to the recursive computations in (9)-(10) and (11)-(12) for hidden layer and output vectors, which is common to all biologically plausible recurrent source separation networks (see Appendix F).
[-• D 1 = 
-]
[-I , and D 
-]
[-• D 
-]
[-I , and D 
-]
[-• M H = 
-]
[-• D 1 = 8 I , and D 
-]
This could perhaps be addressed by early-stopping the recursive computation [60].
[-• M H = 
-]
[-• D 1 = 
 I , and D 
 = I . • µ D 1 = 15 ,-]Acknowledgments and Disclosure of Funding
This work/research was supported by KUIS AI Center Research Award.
C. Pehlevan acknowledges support from the Intel Corporation.