Similarity-preserving Neural Networks from GPLVM and Information Theory
Abstract
This work proposes a way of deriving the structure of plausible canonical microcircuit models, replete with feedforward, lateral, and feedback connections, out of information-theoretic considerations.
The resulting circuits show biologically plausible features, such as being trainable online and having local synaptic update rules reminiscent of the Hebbian principle.
Our work achieves these goals by rephrasing Gaussian Process Latent Variable Models as a special case of the more recently developed similarity matching framework.
One remarkable aspect of the resulting network is the role of lateral interactions in preventing overfitting.
Overall, our study emphasizes the importance of recurrent connections in neural networks, both for cognitive tasks in the brain and applications to artificial intelligence.
Introduction
The idea that the brain is an information-processing entity and its microcircuits deal with separating signal from noise pervades cognitive science, in general, and systems neuroscience, in particular.
It is then natural that, for decades, there have been efforts to use information theory (IT) [1, 2, 3, 4, 5, 6] for explaining psychophysical observations and direct measurements of neural activity.
On the other hand, efforts directed toward deriving learning rules for neural networks (NN) based on informationtheoretic problems [7, 8, 9, 10, 11] need to be examined for biological plausibility [12, 13] before one hopes to posit such NNs as candidates for one of the brain’s canonical microcircuits.
In this work we take up a concrete and simple problem of compressed representation learning [14] via linear Dimension Reduction (DR) [15].
The problem of constructing biologically plausible
NNs from similarity-preserving objective functions, i e., connecting Gramians of inputs and that of corresponding neural activities, has been addressed recently [16, 17, 18, 19, 20, 21].
We keep many of the attractive aspects of the framework but derive a NN and its learning rules starting from a probabilistic formulation known as Gaussian Process Latent Variable Model (GPLVM)[22].
Using
GPLVM, we express the objective function in terms of Gramians but also relate it to the IT criterion that, within some constraints, neural codes are to be maximally predictive of the inputs.
In Section 2, we review the IT criterion and focus on the likelihood of a conditional distribution.
In this section, we also touch upon GPLVM, which optimizes a model-averaged version of the same likelihood.
We argue that, in the large data limit, both these methods recover the true generative model.
In Section 3, we then derive the neural dynamics and the learning rule from a min-max formulation of the GPLVM optimization, with neural implementation discussed in Section 4.
Finally, we note and discuss in Section 5 the fact that the learning rule for lateral interaction between neurons is very different from the one obtained in the similarity matching framework, where it fosters diversity of response by inhibitory interaction between neurons.
We show that, in the probabilistic model, this lateral interaction between neurons essentially leads to redundancy and feature selection, reducing the potential for overfitting. [Table] [Table]
Background and related works
In this section, we introduce the necessary background on information theory and its formulation when ones considers the task of compression [23, 24].
We make the connection with GPLVM [22], a popular framework for dimensionality reduction.
In particular, we discuss the large data limit in which both these methods recover the true generative model.
2.1 Encoding inputs
Let ( X, Y ) be two random variables.
The mutual information (MI) I ( X ; Y ) could be written as
[Equation]
where H ( Y ) :
= − (cid:82) dy p Y ( y )
ln p Y ( y ) and H ( Y | X ) :
= − (cid:82) dxdy p ( x, y )
ln p Y | X ( y | x ) are respectively the entropy of the variable Y and is its conditional entropy, given X .
In this work, we interpret Y as the sensory inputs and X as the brain’s internal encoding of that input.
Since the input distribution is exogenous, the task of mutual information maximization is the same as the task of minimizing the conditional entropy above, subject to constraints (e.g., dimensionality of X ).
In other words, given the encoding, we want, on average, to be the least “uncertainty” regarding the input.
Let { y t } Tt =1 be sampled i.i.d. from p Y ( · ) and x t ∼ p X | Y ( ·| y t ) , p X | Y being the ideal encoder.
Then
[Equation]
where Y =
[ y 1 . . .
y T ] ⊤ refers to n -dimensional observations and X =
[ x 1 . . .
x T ] ⊤ , to the mdimensional encodings.
We make the dependence on the true generative model explicit in this expression p ( Y | X , Model) , in contrast to the model averaged conditional distribution below.
Agakov and collaborators developed an MI-based variational formulation for learning embeddings from a finite-size sample, { y t } Tt =1 , in which they maximize the logarithm of the variational conditional likelihood (cid:80)
Tt =
1 ln q Y | X ( y t | x t ) over an encoder p X | Y , and a variational decoder q Y | X [23, 24].
We, instead, turn to a Bayesian approach for latent variable discovery, dealing with the likelihood involving only { y t } Tt =1 and { x t } Tt =1 , obtained by integrating over potential generative model parameters.
2.2 Generative models and probabilistic linear DR
The generative model for linear DR we consider is the following.
We assume that n -dimensional observations Y are a linear function of m -dimensional latent variables
X [25, 22], i.e., [Equation]
The linear map A ∈ R n × m relates the two sets of variables.
Furthermore, in GPLVM [22] a prior is imposed on the model parameters A , unlike for probabilistic PCA [25], as
[Equation]
Since the conditional distribution Y | X is linear in A and [ ε 1 , . .
. , ε T ] , we can replace our original proposal of the objective function, Eq.(2) , p ( Y | X , Model) :
=
p ( Y | X , A ) by taking an expectation over A with the GPLVM prior, Eq.
(4), namely p ( Y | X ) :=
E A [ p ( Y , A | X )] .
We recall the conditional for samples of each component of y , namely y ( d ) =
[ y d 1 , . . .
, y dT ] in GPLVM as
[Equation]
The model averaged distribution of Y | X for GPLVM, in terms of the Gramians, X ⊤ X , Y ⊤ Y , is
[Equation]
where | · | denotes the matrix determinant.
In the limit of large T , for a given Y , the variational procedure mentioned in Section 2.1, carried out over a limited class of encoders and decoders parameterized by A would recover the true A with which the data was generated [24].
In the Bayesian approach leading to the GPLVM objective, given Y , Bernstein-von Mises type theorems [26, 27, 28] state that the posterior distribution of A will be concentrated around the true A , as T → ∞ .
Therefore, the averaging over A , for the high probability X , approximates picking the optimal/true A .
A Min-max objective function for GPLVM
While the GPLVM objective (6) can be minimized by gradient descent [29], this procedure would not lead to an online algorithm as it requires combining data from different time steps.
Instead, following the work of [17], we modify the objective (6) by introducing auxiliary matrix variables, W , M , and
Z allowing for the GPLVM computation using solely instantaneous inputs.
Such substitution leads to a min-max optimization problem that is solved by gradient descent/ascent and maps onto an NN with local learning rules presented in Sec. 4.
We now introduce auxiliary matrix variables W and M .
To do so we first address the term that only depends on X ,in Eq. (6), i.e., the log-determinant term (more details in Appendix A)
[Equation]
Now by using the Woodbury matrix identity to the cross-term involving both X and Y , in Eq. (6), and by only keeping the term that depend on X we obtain
[Equation]
It is important for the neural interpretation that we introduce another auxiliary variable Z into Eq.
(8) that will account for the projection error as
[Equation]
Now we combine Eq.(7) and Eq.(9) to obtain a min-max objective function for linear GPLVM as
[Equation]
Online Algorithm and neural implementation
From the min-max objective function Eq.
(10) we first derive an online algorithm and then present its implementation by a biologically inspired neural network.
Gradient optimization.
We first optimize Eq.
(10) with respect to the “neural variables” x t and z t , for fixed W and M , by gradient descent/ascent.
At each time step, t , we repeat the following gradient descent ascent steps until convergence:
[Equation]
Then, for fixed x t and z t , we minimize the objective function over W and M , take stochastic gradient descent-ascent steps, which yields
[Equation]
This yields our online GPLVM algorithm (Algorithm 1) and the neural implementation Fig.
Neural implementation.
The algorithm for online GPLVM (Algorithm 1) summarized by the dynamics Eqs.
and update rules in Eqs.
(12) can be implemented in a neural circuit with schematic shown in Fig.
In this circuit, the individual components of the output { x 1 ,t , . . .
, x m,t } , are represented as the outputs of m neurons, with M the lateral synaptic connections between them.
The auxiliary variable z t is represented by the activity of m inter/hidden neurons with W encoding the connection between z t and x t .
In a biological setting, the implied equality of weights of synapses from z t to x t and the transpose of those from x t to z t can be guaranteed approximately by application of the same Hebbian learning rule [30, 31].
Another possible way to interpret the model would be as y t and z t being respectively the somatic and axonal terminal activities of two-compartment unit neurons as in [32].
Update rules requiring computing weight matrix inverse are also present for tasks such as independent component analysis [33] for which local alternatives exist [10].
input data { y 1 , . . .
, y T } ; dimension n output { x 1 , z 1 , . . .
, x T , z T } ; dimension m and n ▷ estimated embedding and error initialize the matrix W , and positive definite matrix M .
for t = 1 , 2 , . . .
, T do z t ← y t ; ▷ first error no estimated x t run the following until convergence: d x t ( γ )
[Equation]
; ▷ output dynamics
[Equation]
=
[Equation]
; ▷ error dynamics W ← W + η z t x ⊤ t ; ▷ Feedforward weight updates
M ← M + ητ ( σ [-
-]2 I m + x t x ⊤ t − M − 1 ) ; ▷ Lateral weight updates end for - M
[Figure] [Figure]
Numerical experiments.
As an illustration of the capability of the NN derived from GPLVM, we provide experimental results of our algorithm.
We evaluate our algorithm on an artificially generated datasets, X ∈ R 20 × 10 , 000 , with a linear spectrum, ( λ j = j/ 10 , ∀ j ∈ { 1 , 20 } ), shown in Fig.2.
The performance of our algorithm is measured based on the subspace alignment error, i.e., the difference in norm square with normalized projector obtained by PCA and the one encoded in ( W , M ) .
In
Fig.
2, we show that after convergence, our algorithm leads to a projection on the true basis vectors for different output dimensions.
Discussion
In this work, we asked the question of the implication of IT as an organizing principle for cognition and understanding of the brain.
More precisely, how does the brain learn from input distributions that it has not seen before or in novel perceptive environments?
Nonetheless, inspired by work on similarity-preserving NNs (recalled in Appendix B), we proposed a novel min-max objective function that gives rise to a neural implementation for GPLVM.
Our work illustrates the point by discussing linear GPLVM, but this could be extended to other kernels easily, using random features [34, 35].
We observe, however, two striking differences with non-IT-based NNs.
First, the intermediate variables z t play the role of error computing neurons rather than projection neurons.
As a result, the update of the feedforward weights, W , (Eq. (12)Left) minimizes the prediction error rather than maximizing the correlation of the projected input with the inputs.
Secondly, the learning rule in M , (Eq.
(12)Right) involves the matrix M − 1 , which in turn increases the connection between correlated output (as detailed in Appendix C).
It is unlike in more standard models where lateral connection aims at decorrelating output activities through means of anti-Hebbian updates [17].
More precisely, in IT, the lateral connections M prevent overfitting, while in SM, it tries to avoid underfitting.
This raises the question of how we can probe such an observation experimentally and favor IT or non-IT-based frameworks. Acknowledgment
The authors would like to thank the members of the Neural Circuits and Algorithms for helpful comments on this manuscript.
AMS’s work was partly supported by an award from the Simons
Foundation (SF626323).
Appendix A Detailed min-max steps
We provide below more details on how to obtain the min-max objective function Eq.
We start from the log-likelihood for GPLVM Eq.(6) recalled below
[Equation]
We now introduce auxiliary matrix variables W and M .
To do so we first address the term that only depends on X ,
i
[Equation]
e., the logdet term
[Equation]
Now the cross-term involving both X and Y using Woodbury matrix identity and by only keeping the term that depend on X we obtain
[Equation]
It is important for the neural interpretation that we introduce another auxiliary variable Z into Eq.
(8) that will account for the projection error as
[Equation]
Now we combine Eq.(7) and Eq.(9) to obtain a min-max objective function for linear GPLVM as min
[Equation]
B Similarity matching for non-probabilistic linear DR
In order to derive a NN from GPLVM, we look at the similarity matching (SM) framework [17], which proved able to derive such NNs for non-probabilistic similarity-preserving objective function.
SM has the following objective function, written in terms of the Gramians X ⊤ X and Y ⊤ Y :
[Equation]
The work on SM took a different approach than that of Oja [36] on PCA, as detailed in [12], and showed that the resulting rules were strictly local unlike Oja’s rule.
They solve Eq.
(18) by introducing a min-max objective function and auxiliary variables as
[Equation]
The objective (19) can be optimized by an online algorithm that maps onto a NN whose synapses obey local learning rules [17].
However, it is unclear how uncertainty can be incorporated and thus produce an IT equivalent of SM for deriving NNs.
We address this shortcoming by deriving NNs from the GPLVM objective function instead.
C Details on the lateral connections ( M ) update rule
The x t dynamics rule:
[Equation]
says that the sign of the lateral interaction between output neuron a , whose activity is x at ( γ ) and output neuron b ( b ̸ = a ), whose activity is x bt ( γ ) , is decided by the sign of M ab .
If M ab is positive, it is inhibitory, if M ab is negative, it is activating.
We want to argue that as new neural activities (latent variables) come in, providing some signal of additional correlations, M learning amplifies these correlations.
Namely, empirical extra positive correlation leads to excitatory interaction while negative correlation leads to inhibitory one.
The offline M optimality condition is M =
( σ 2 I m + 1 T (cid:80)
Tt =1 x t x ⊤ t ) − 1 .
Our learning rule approximately tries to achieve this equality.
Now let us see how later entries affect this quantity.
Let us split
T observations into the first T 0 observations and the batch of the last B observations, with the batch size B ≪ T .
{+Now define M 0 as M 
+}
=
( σ 2 I m +
[Equation]
0 (cid:80) T 0 t
=1 x t x ⊤ t ) − 1 .
With this definition, one can check that
[Equation]
Since B ≪ T we have
[Equation]
Imagine now that we choose a basis so that (cid:80) T 0 t =1 x t x ⊤ t
and, therefore, M 0 is diagonal.
Then, for a ̸ = b ,
[Equation]
where { µ 0 a } ma =1 are the eigenvalues of M 0 .
Since these eigenvalues are positive for active channels, recent signature of correlation between channels a, b affects M ab with opposite sign, leading to our correlation amplification effect.
On the whole, it cuts down unnecessary latent variable directions, which would mostly represent noise in the input.
The equivalent argument for similarity matching [37] produces, with a crucial sign difference,
[Equation]
What then maintains the diversity of channels for GPLVM?
If some component of x t becomes too small so that its explanatory power for y t reduces, the error variable z t becomes large and provides the countering feedback via W update.
This also stands totally in contrast to similarity matching [37].