Translation-equivariant Representation in Recurrent Networks with a Continuous Manifold of Attractors
Abstract
Equivariant representation is necessary for the brain and artiﬁcial perceptual systems to faithfully represent the stimulus under some (Lie) group transformations.
However, it remains unknown how recurrent neural circuits in the brain represent the stimulus equivariantly, nor the neural representation of abstract group operators.
The present study uses a one-dimensional (1D) translation group as an example to explore the general recurrent neural circuit mechanism of the equivariant stimulus representation.
We found that a continuous attractor network (CAN), a canonical neural circuit model, self-consistently generates a continuous family of stationary population responses (attractors) that represents the stimulus equivariantly.
Inspired by the Drosophila’s compass circuit, we found that the 1D translation operators can be represented by extra speed neurons besides the CAN, where speed neurons’ responses represent the moving speed (1D translation group parameter), and their feedback connections to the CAN represent the translation generator (Lie algebra).
We demonstrated that the network responses are consistent with experimental data.
Our model for the ﬁrst time demonstrates how recurrent neural circuitry in the brain achieves equivariant stimulus representation.
Introduction
Neuronal responses in the brain internally represent the stimulus s in the world.
When the stimulus changes due to some transformations, the brain updates its internal representation accordingly to faithfully reﬂect the stimulus change.
For example, V1 neurons encode the visual orientation, and their population responses track the changing orientation which enables subsequent brain areas to decode the instantaneous orientation.
Transformations acting on a continuous stimulus can be described by the Lie group theory [1], and a neural representation reﬂecting the stimulus transformation is called group equivariant representation [2–4].
Mathematically, supposing a Lie group G acting on a continuous stimulus s , the evoked neural population response ¯ u ( s ) (a continuous function of s , and ¯ u denotes the mean response) is equivariant with the action of group G (group homomorphism ) if,
[Equation]
Stimulus feature s
M ean r e s pon s e
M ean r e s pon s e u ( x { s ) u
[ x { ( s + a )]
Neuron index x s s + a
T ( a ) ^
T g ¢ ^
T g ¢ ¹ ¹ u ( x j { s ) u u ¹ u ( s ) ¹ u ( s 0 ) ¹
C B A s s
Equivariant map
Stimulus
Neuralresponse
ˆ T g and T g are the induced actions (operators) with group element g ∈ G which respectively act on neuronal responses and the stimulus, and · denotes the action of group operators.
Intuitively, the equivariant neural representation (Eq. 1) implies the neuronal responses evoked from a transformed stimulus (e.g., rotating the orientation of Gabor images), i.e., ¯ u ( T g · s ) (Fig.
1A, blue path) would be the same as transforming the neuronal responses evoked by the original stimulus along the stimulus manifold, i.e., ˆ T g · ¯ u ( s ) (Fig.
1A, green path).
The equivariant representation is indispensable for the brain to represent the stimulus under transformations.
An equivariant neural circuit is not only required to represent the stimulus equivariantly ( ¯ u ( s ) , Eq. 1), but it also needs to explicitly represents the Lie group operators ( ˆ T g , Eq. 1).
Explicitly representing group operators enables the brain to actively synthesize a transformed stimulus and predict the changing stimulus under transformations [5].
For example, when tracking a moving object, the brain not only represents the object’s instantaneous position s , but also represents its moving speed (the translation group parameter) in order to predict the stimulus position in the next time step.
Yet it is an open question as to how recurrent neural circuits in the brain equivariantly represent the stimulus, and how abstract group operators are represented in the brain.
Neuroscientists utilized the Lie group to investigate the perception and representation in the brain (e.g., [6–14]), however, nearly none of previous studies linked the abstract group theory to concrete biological recurrent circuit dynamics.
Meanwhile, although equivariant neural networks have been extensively studied recently in machine learning (ML) research to capture transformations of stimuli (input data) deﬁned on graphs (e.g., [15]) or deﬁned on smooth manifolds (e.g., [3, 4, 16, 17]), nearly all these studies focused on developing feedforward weights which act as group convolution kernels in order to yield equivariant outputs under group transformations.
There has not been much ML research studying equivariant recurrent networks, nor explicit network representations of group operators.
To provide theoretical insight into this fundamental question, we used a 1D translation group acting on a 1D stimulus feature as an example to investigate the general recurrent circuit mechanism of equivariant representation.
We found the equivariant stimulus representation can be naturally realized in continuous attractor networks (CANs) which self-consistently generate a continuous family of stationary population responses (attractors) to represent the stimulus equivariantly.
Moreover, the translation group operators are represented by separate neuronal populations whose responses represent the translation speed (translation group parameter).
The feedback connections from speed neurons to CANs are shifted on the stimulus manifold, which encode the translation generator (Lie algebra).
We compare the proposed recurrent circuit model with Drosophila’s internal compass circuit to show its biological plausibility.
The present study is one of the ﬁrst studies linking the equivariant representation to a biologically plausible neural circuit model with rigorous mathematical analysis.
It will shed light on the neural representations in the brain.
Translation group and homogeneous neural population code
We start from a concrete Lie group and neural representation to investigate the general circuit mechanism of equivariant representation.
As an example, we consider a 1D translation Lie group T acting on a 1D stimulus s which can be regarded to be heading direction, orientation, moving direction, spatial location, etc.
A group element T ( a ) ∈ T translates the stimulus s to s + a , i.e.,
T ( a ) · s = s
+ a , and therefore an equivariant neural representation ¯ u ( s ) should satisfy (Fig. 1C),
[Equation]
where ˆ T ( a ) is the 1D translation operator inducing the translation of neural representation (Fig. 1C).
Several properties of the ˆ T ( a ) can be directly derived from the deﬁnition (Eq. 2) [18],
[Equation]
First, a translation operator with zero translation should leave the neural representation of stimulus to be the same as no transformation (Eq. 3a).
Second, the translation operator can be composed together and the composition is not dependent on the order of operators (Eq. 3b), which implies the translation operator is commutative or abelian [18].
Third, the inverse of a translation operator corresponds to an operator translating to the opposite direction with the same distance (Eq. 3c).
Since the translation is continuous, the amount of translation can be made inﬁnitesimally small.
Consider an inﬁnitesimal translation δa → 0 on the stimulus s , then the corresponding change of neural representation is,
[Equation]
where ∂ s =
∂/∂s denotes the derivative over s , and a ﬁrst order Taylor expansion was used in above derivation.
ˆ p ≡ ∂ s is called the translation generator characterizing the tangential direction of translation in the group space, and forms a basis of the Lie algebra .
In physics, ˆ p is usually regarded as the momentum operator [18].
Using the inﬁnitesimal translation, the differential form of a translation operator can be derived as (see details of Lie group in Supplementary Information (SI)),
[Equation]
By exponentiating the two sides in Eq.
(5), the translation operator can be solved as ˆ T ( a ) =
exp( a ˆ p ) , which is an exponential map of the translation generator ˆ p .
We next deﬁne a concrete neural representation based on experimental observations.
The neural representation ¯ u ( s ) =
{ ¯ u j ( s ) } Nj =1 is regarded as the mean responses of a population of N neurons.
Since the translation operator (Eq. 2) acts on a continuous function of neural population response, i.e., ¯ u ( s ) , we therefore consider the limit of a large neuronal population with inﬁnite number of neurons ( N → ∞ ), and then ¯ u ( s ) effectively converges to a continuous function (a vector of inﬁnite dimension in Hilbert space).
Inspired by the bell-shape stimulus tuning curves widely observed in the cortex, the mean response of the j -th neuron is modeled as a Gaussian function of the stimulus s (Fig.
1B),
[Equation] [Equation]
) where x j denotes the preferred stimulus of a neuron, indicating the neuronal response reaches its maximum when the stimulus matches its preferred value, i.e., s = x j .
U j and σ j denote respectively the peak ﬁring rate and the tuning width of the j -th neuron.
A necessary condition to construct an equivariant neural population code is homogeneous neurons in the population, which means the tuning curves of all neurons have the same proﬁle, i.e., U j ≡ U and σ j ≡ σ for any j , and the preferred stimulus of all neurons, { x j } , is uniformly distributed on the stimulus space.
In this way, the mean responses of all neurons in response to a stimulus s are (Fig. 1C)
[Equation]
which can be veriﬁed to satisfy the translation equivariant representation (Eq. 2).
Towards an equivariant recurrent neural circuit model
We next investigate how the equivariant homogeneous neural code, i.e., ¯ u ( x − s ) in Eq.
(7), can be self-consistently generated by a concrete recurrent neural circuit dynamics.
Speciﬁcally, we explore the recurrent circuit mechanism to generate a stable stimulus representation ¯ u ( x − s ) without translation (corresponding to a zero translation operator, ˆ T (0) ).
Afterward, we investigate the neural representation of translation operators and how it induces the translation of stimulus representation.
A
[Figure]
C
B
D
F E
[Figure]
3.1 The autonomous dynamics of continuous attractor networks
Since the recurrent neural circuit dynamics needs to generate non-zero mean responses ¯ u ( x − s ) for any value s (Eq. 6), the neural dynamics should be stable at any s .
That is, there is a manifold of ﬁxed points (attractors) in the state space of network dynamics.
The continuous manifold of attractors can emerge from the autonomous dynamics (no external input drive) of continuous attractor networks (CANs, Fig.
2A), which is a canonical network model explaining the processing of continuous stimuli in neural circuits [19, 20], including heading-direction [21], orientation [22], spatial location [23] and so on.
In the present study, we consider a CAN dynamics as follows [24, 25],
[Equation]
8b) where u ( x, t ) and r ( x, t ) denote respectively the synaptic input and ﬁring rate of the excitatory neuron preferring stimulus s = x at time t , and τ is the time constant of synaptic input.
G ( · ) is an activation function that is modeled as divisive normalization where [ · ] + is nonnegative rectiﬁcation and k determines the global inhibition strength.
The divisive normalization is a canonical operation widely observed in the cortex [26], and is resulted from the parvalbumin (PV) inhibitory neurons [27].
W r ( x, x (cid:48) ) denotes the recurrent weight from the neuron preferring x (cid:48) to the neuron preferring x , which is modeled as a Gaussian function of the disparity between the preferred stimuli of the preand post-synaptic neurons (Fig. 2B), i.e.,
[Equation]
(9) where w r is a scalar and determines the peak recurrent weight, and σ the width of connection in the stimulus space.
Because we consider a large population of neurons that are uniformly distributed in the stimulus space, the neuronal interactions effectively become an integral in Eq.
8a, with ρ denoting the neuronal density in the stimulus space.
Furthermore, the translation-invariant recurrent connections, which only depend on x − x (cid:48) , imply that the recurrent interactions are effectively conducting convolution on the stimulus space, i.e., (cid:82) W r ( x − x (cid:48) )
· r ( x (cid:48) , t ) dx (cid:48) ≡ ( W r ∗ r )( x, t ) , where the symbol ∗ denotes the convolution over x .
3.2 A continuous attractor network equivariantly represents the stimulus
It can be checked that the CAN dynamics (Eqs. 8a-9) can generate a family of non-zero Gaussianproﬁle stationary responses that are consistent with the proposed responses (Eq. 6) [25, 28],
[Equation]
as long as the recurrent weight is stronger than a critical value which is solved as w c = 2 √ 2(2 π ) 1 / 4 (cid:112) kσ/ρ (see SI.
Sec. 2).
It is worth noting that s is a free parameter in Eq.
(10), indicating that the network has a continuous family of ﬁxed points, i.e., ¯ u
( x − s ) , which selfconsistently emerge from the recurrent neural dynamics without the inﬂuence from external inputs (Eqs.
8a-8b).
The stationary population responses (Eq. 10) are consistent with the equivariant homogeneous neural population code (Eq. 7), and therefore they represent the stimulus equivariantly.
The stationary population responses in the recurrent network should be stable in order to maintain the equivariant stimulus representation robustly, otherwise, the population responses can be easily disrupted by noises.
Therefore we check whether the continuous family of population responses (Eq. 10) are stable (attractors), by performing the stability analysis of the network dynamics.
We perturbed the network state around the ﬁxed points by adding a small perturbation δu ( x, t ) , i.e., u ( x, t ) =
¯ u ( x − s )
+ δu ( x, t ) , and then the dynamics of the perturbation is derived as [28],
[Equation]
The interaction kernel K ( x, x (cid:48) | s )
=
ρ
[Equation]
W r ( x − x (cid:48)(cid:48) )
∂ ¯ r ( x (cid:48)(cid:48) − s ) /∂
¯ u ( x (cid:48) − s ) dx (cid:48)(cid:48) , whose analytical expression can be seen at SI.
Sec.
Treating K ( x, x (cid:48) | s ) as an operator acting on the perturbation δu ( x, t ) , all of its eigenvalues, λ n , and (unnormalized) eigenfunctions, f n , can be analytically calculated (listed with a descending order) [28],
[Equation]
The explicit expression of eigenfunctions for n ≥ 3 can be found at SI.
Sec.
We see no eigenvalues are larger than 1 (Fig.
2F) indicating that the continuous family of ﬁxed points ( ¯ u ( x − s ) in Eq. 10) are stable, i.e., the network has a continuous family of attractors, and each attractor is an equivariant representation of the stimulus s (Fig. 2E).
Representing translation operators in CANs
It is noteworthy that the eigenfunction f 1 ( x | s ) of the perturbed network dynamics (Eq.
2F, top) is parallel to the direction of translation generator (Eq. 12a), i.e., f 1 ( x | s )
∝ ∂ s ¯ u ( x − s )
=
ˆ p · ¯ u ( x − s ) , thus f 1 ( x | s ) corresponds to the translation of neural responses along the continuous attractor manifold representing the stimulus s (Eq.
2F, top).
Meanwhile, its corresponding eigenvalue λ 1 = 1 suggests the eigenfunction f 1 ( x | s ) is neutrally stable, which is resulted from the translation invariance of recurrent connections in the network (Eq. 9).
Nevertheless, although the CAN dynamics maintains the translation operator resulted from the external perturbation δu ( x, t ) , it does not self-consistently support non-zero translation generators in that the stationary network states are static without any translation (Eq. 10, Fig.
2C-D), i.e., corresponding to ˆ T (0) .
We next explore a self-sustained network mechanism to represent non-zero translation operators, i.e., ˆ T ( a ) , ( a (cid:54) = 0)
(Eq. 2), in order to induce translations of neuronal responses ¯ u ( x − s ) .
4.1 Incorporating translation operators in a CAN
To embed translation operators in the CAN dynamics, we derive a differential equation of Eq.
and compare it with the CAN dynamics.
We consider the translation has instantaneous speed v ( t ) , which determines the translation amount in a time window dt as da = v ( t ) dt .
Taking the derivative of Eq.
W r ( x , x 0 ) : support non-zero response u ( x { s )
A
B
C
D    10 Time t ( ¿ ) 5N eu r on i nde
x xB
u
m p
po
s i
t i onr ( t )
Speed v =
Speed (theory) 50S peed ( ne t w o r k ) ¹Speed
SimTheoryS peedI n t eg r a t ed s t i m u l u s 10 Time t ( ¿ )
Commutation @
x W r ( x , x 0 ) : induce translation on stimulus manifold
Tuning disparity x − x '
R e c u rr en t w e
i
gh t  w r ¿v
F i r i
ng r a t e ( a . u . )
(2) over time t , and utilizing the differential form of translation operators (Eq. 5), we derive that (to simplify notation we set s = 0 without loss of generality),
[Equation]
u ( x − a ) .
Replacing the variable x − a by x , we reach the differential equation governing the translation of neuronal responses with speed v ( t ) ,
[Equation]
In comparison, the CAN dynamics in the stationary state is d ¯ u ( x ) /dt
=
(Eq. 8a), i.e., the network state is static.
To incorporate the translation operator with speed v ( t ) (Eq. 13), we utilize that the stationary CAN state satisfying ¯ u ( x − s )
=
ρ W r ∗ ¯ r ( x − s ) , and substitute it to the right hand side of
Eq.
[Equation]
where we have used ∂ s = − ∂ x (Eq. 7), and swapped the convolution and partial derivative.
The above equation provides an insight that the translation generator can be converted into the derivative of recurrent connections in the network, i.e., ∂ x W r .
Therefore, it motivates us to consider a modiﬁed
CAN dynamics whose recurrent connections have a new component ( ∂ x W r , Fig.
3A) [21, 29],
[Equation]
Next we verify whether the modiﬁed CAN dynamics (Eq. 15) embeds translation operators, i.e., a spontaneously moving neural response along the stimulus manifold with speed v .
We substitute the mean population response ( ¯ u ( x − s ) and ¯ r ( x − s ) , Eq. 10) into the above network dynamics, repeatedly use the stationary response that satisﬁes ¯ u ( x − s )
=
ρ W r ∗ ¯ r ( x − s ) to cancel terms,
[Equation]
Comparing Eqs.
(16) and (13), we see the stationary state in the modiﬁed CAN spontaneously translates along the stimulus manifold in a way consistent with the action of translation operators (Eq. 13).
Therefore the recurrent connection component ∂ x W r represents the translation generator, and its magnitude determines the moving speed v .
Numerical simulations of the network model also conﬁrm our theoretical derivations (Fig.
3B-C), where the actual moving speed is well predicted by the theory using the magnitude of recurrent connection ∂ x W r .
When the network response deviates from the stereotypical Gaussian proﬁle (Eq. 10), the recurrent connection component W r will remove distortions because the eigenvalues of all perturbations perpendicular to the continuous stimulus manifold are smaller than 1 (Eq. 12a - 12c).
Once the population response settles into a smooth Gaussian proﬁle, the network response translates along the continuous manifold with the desired speed (see details in SI.
Sec. 4).
4.2 Path integration in the modiﬁed CAN dynamics
The brain accumulates the effects of a series of translation operators, and the combined translation effects are not dependent on their action orders (Eq. 3b), i.e., translations are commutative.
The translation accumulation that is independent of action orders corresponds to the path integration in the neuroscience research [23, 29, 30], where the animals’ navigation system integrates their walking path to update their walking distance.
The path integration can be achieved in the modiﬁed CAN dynamics (Eq. 15).
To illustrate this, we compute the translation distance in the modiﬁed CAN dynamics (Eq. 16) by using the chain rule, ∂ ¯ u ( x − s
[Equation]
∂t
[Equation]
v
[Equation]
, and substitute it back to Eq.
Finally, the translation distance is derived as
[Equation]
(17) where ds is the translation distance along the continuous stimulus manifold in the CAN in a time window lasting dt .
As an example, we consider a constant moving speed v 1 lasts for time duration t 1 , followed by another constant speed v 2 lasting for duration t 2 , and then we swap the order of two speeds (Fig.
3D, top).
The moving distance in the two cases can be calculated as the same,
[Equation]
18) which shows the translation in the modiﬁed CAN is commutative, i.e., the network achieves path integration.
This is also conﬁrmed by numerical simulations (Fig.
3D, bottom).
A biologically plausible equivariant recurrent circuit
Although in the modiﬁed CAN dynamics the recurrent connection component ∂ x W r
(Eq. 15) mathematically represents the translation generator and translates network responses, the connection ∂ x W r is not biologically plausible due to two reasons.
First, the magnitude of the connection component ∂ x W r needs to be multiplicatively modulated by the speed v (Eq. 15), but multiplicative modulation of synaptic weights is challenging in neural systems.
Second, with Gaussian-proﬁle connections (Eq. 9), the derivative ∂ x W r means that a neuron has both excitatory and inhibitory synapses, which disobeys Dale’s law that a neuron can be either excitatory or inhibitory but cannot be both.
Therefore, we propose a new neural circuit model to represent the translation operator that is consistent with the biological neural circuitry.
Mathematically, we will replace the term of multiplicative synaptic weight modulation (Eq. 15, last term) by a biologically plausible input term that is originated from separate neuronal populations whose responses represent moving speed v .
The new circuit model is inspired by the Drosophila’s compass system which consists of three neuronal populations (Fig.
4B, e.g., [31, 32]).
There is a ring of heading-direction (stimulus s ) neurons whose dynamics is governed by a CAN (Fig.
4B, E-PG neurons in blue disk, [33]).
Besides, there are two populations of speed neurons (Fig.
4B, left and right P-EN neurons) whose ﬁring rates encode the speed (translation distance a in a unit time).
Moreover, the feedback projections from speed neurons to heading-direction neurons are either clockwise or anti-clockwise shifted and they induce translations of heading-direction neurons’ responses.
Therefore, our new recurrent circuit model has three populations of neurons (Fig.
4A): the stimulus s neurons which is modeled as a
CAN as before (Eqs.
8a-8b), and two populations of speed neurons which are supposed to encode the positive and negative speed respectively.
The network dynamics is governed by [31, 34–36],
[Equation]
19b
[Equation]
Eq. (19a) is the CAN dynamics of stimulus neurons.
u ± ( x, t ) ( r ± ( x, t ) ) are the synaptic inputs (ﬁring rate) of speed neurons which represent positive and negative speeds respectively and are driven
[Figure]
C
[Figure]
(G)
The proposed network implements the path integration with consistent speed as theory.
by the stimulus neuron r ( x, t ) preferring stimulus x .
Note that stimulus neurons receive feedback inputs from speed neurons (Eq. 19a, last term), which replaces recurrent inputs from biologically implausible recurrent weight ∂ x W r
(Eq. 15, last term).
Consistent with the Drosophila’s compass [31, 32, 34], recurrent connections W ± ( x, x (cid:48) ) from the speed neuron x (cid:48) to the stimulus neuron x is shifted by ∓ ∆ x towards the purported moving direction encoded by speed neurons (Fig. 4C),
[Equation]
Moreover, speed neurons’ ﬁring rates r ± ( x, t ) are multiplicatively modulated by the speed v (Eq. 19b).
This operation is necessary to reproduce gain modulation of speed neurons’ responses (speed invariant stimulus tuning width) in present rate-based model [31], and it achieves the same computation with multiplicative modulation of synaptic weights (Eq. 15, last term).
g v is a constant determining the baseline activity of speed neurons.
To emphasize the main computational mechanism, we simplify the circuit model (Eqs. 19a-19b) compared with the Drosophila’s circuit.
First, we ignored the interactions between speed neurons.
Second, we assumed that a speed neuron is only driven by a stimulus neuron with the same preferred stimulus, which is exhibited by the scalar w vs (Eq. 19b).
5.1 Neural representation of translation operators
We investigate how the abstract translation operator ˆ T ( a ) is represented in the circuit model inspired by the Drosophila’s compass.
Similar to the theoretical analysis performed in Eqs.
(15-16), we focus on the latent network dynamics along the dominant continuous stimulus manifold by ignoring the ﬂuctuations along other directions.
Then Eq.
(19a) can be simpliﬁed into (see more details in SI. 5),
[Equation] [Equation]
where R + and R − are the peak ﬁring rates of positive and negative speed neurons, and U is the peak synaptic input received by stimulus neurons (Eq. 10).
The shifted connections W ± ( x, x (cid:48) ) from speed neurons to stimulus neurons introduce the translation generator, in that the magnitude of translation generator is proportional to the connection shift ∆ x (Eq. 16).
The excitatory W ± ( x, x (cid:48) ) pulls the network response towards shifted directions (implementing the excitatory part of the translation generator), while the inhibitory part of the translation generator is achieved through the network effect of global inhibition (Eq. 8b).
Moreover, the ﬁring rate difference between two populations of speed neuron, R + − R − , determines the translation speed (1D translation operator parameter).
Overall, the ﬁxed translation generator ˆ p is stored in the shifted connections from speed to stimulus neurons, and the time-varying translation speed is represented by the ﬁring rate difference between two speed neuron populations which can change in a short time scale.
Meanwhile, the speed neurons’ dynamics (Eq. 19b) suggests R + − R − =
[( g v + v ) − ( g v − v )]
= 2 v w vs R .
To translate network responses with the desired speed, the coefﬁcient on the right hand side of Eq.
(21) should equal to the speed determining speed neurons’ response (Eq. 19b), which requires,
[Equation]
We performed numerical simulation of the circuit model (Eqs. 19a-19b), whose parameters are set to satisfy Eq.
We ﬁrstly veriﬁed our theoretical prediction, i.e., the moving speed is proportional to the ﬁring rate difference between two populations of speed neurons, i.e., R + − R − (Fig.
4D-E).
Indeed, the actual moving speed of stimulus neurons’ population responses is consistent with the required speed that determines the ﬁring rate of speed neurons (Fig. 4G), which supports the validity of our theoretical analysis and indicates the accurate path integration in the proposed circuit model.
The joint tuning curves of two example speed neurons with respect to the stimulus s and moving speed v are plotted in Fig.
4F, which are multiplicatively modulated by the speed, i.e., the speed invariant stimulus tuning width, and this is consistent with experimental observations [31].
Conclusion and Discussion
Equivariant representation is necessary for the brain and artiﬁcial perceptual systems to reliably represent the stimulus under transformations [2–4, 6, 7, 37].
The present study investigates how recurrent neural circuits implement equivariant stimulus representation and Lie group operator representation.
Our theoretical analysis shows that the stimulus can be equivariantly represented in a homogeneous neural population code that collectively emerges from a CAN dynamics.
The Lie translation group operators are represented by extra neuronal populations whose activities represent the translation speed (translation group parameter) and whose feedback connections to the CAN store the translation generator.
Our study for the ﬁrst time links the Lie group equivariance to a concrete, biologically plausible recurrent neural circuit.
It gives us insight into understanding the neural coding principle and may inspire a new building block for equivariant representation in ML tasks.
The CAN is a canonical neural circuit model widely used in neuroscience [20, 22, 25, 38, 39].
Recurrent circuit models similar to the present model were proposed before (e.g., [29, 31, 34– 36, 40]), whereas no previous studies theoretically linked the network dynamics to the equivariant stimulus representation, nor the Lie group operator representation.
Besides, our theory also derives the condition that network responses translate with the desired speed (Eq. 22), which is testable by analyzing experimental data.
To highlight the basic mechanism of equivariant representation in a recurrent circuit, the proposed neural circuit model has some simpliﬁcations compared with neural circuits in the brain.
First, we considered homogeneous neurons and ignored the randomness on recurrent connections and neural noises in real cortical circuits.
Apparently, the neural noise and random connections will break the perfect structure of equivariant representation.
It has been suggested that the CAN dynamics is resistant to input noise by removing the noise component that is perpendicular to the attractor manifold [24, 41], and then the equivariant representation can be approximately held.
Another possibility is that the neural heterogeneity and neural noise can help the brain implement sampling-based Bayesian inference (e.g., [42–47]), where the instant response of the CAN and the activity of speed neurons represent the samples of heading direction and rotation speed, respectively (e.g., [48, 49]).
Second, we analyzed a limiting case of inﬁnite number of neurons in a recurrent circuit model, resulting in that the translation operator (Eq. 2) acts on a continuous function of neuronal population response.
Although it brings huge beneﬁt in theoretical analysis, the real neural circuit only has ﬁnite number of neurons, especially there are only dozens of E-PG neurons in Drosophila’s heading direction circuit (Fig.
4A, [31, 32]).
In the case of ﬁnite number of neurons, the recurrent circuit model can still have equivariant representation, however, the proﬁle of population response and the convolution kernel will slightly deviate from Gaussian and the concrete proﬁle can be obtained by numerical optimization, as suggested by a recent work [40].
The proposed neural circuit model can be extended to capture more characteristics in real neural circuits.
For example, our model doesn’t have the sensory input in order to emphasize an internal mechanism inducing the translation of neural representation.
The sensory inputs that represent instantaneous stimulus direction and the instantaneous velocity into Eqs. (19a and 19b) respectively, similar with in earlier results [31, 34].
Moreover, the neural circuit model considers Gaussian proﬁle of population responses, while neuronal tunings sometimes have long tail with high kurtosis [50].
In principle, the recurrent connection proﬁle can be varied to let the recurrent circuit self-consistently generate different tuning curves.
In particular, the high kurtosis proﬁle may be considered a scalemixture of Gaussians, i.e., Gaussians with different scales, and then it can be generated by recurrent connections that is also consist of mixture of many Gaussian connection proﬁles
To elucidate the concepts clearly, the present study only investigates the 1D translation group while the proposed framework can be generalized to more complicated group transformations.
For example, we may develop a recurrent circuit with a 2D translation group equivariant representation to explain the spatial location coding in place and grid cells (e.g., [30, 51, 52]).
A recent study reported that the population activities of grid cells appear to reside in a 2D torus manifold [53], which is the 2D version of the ring that is being studied in the present study.
Meanwhile the CAN based on 2D torus was also proposed in previous studies (e.g., [30, 48]).
And therefore it is possible to generalize our current theoretical framework to the 2D domain to partially explain the properties of grid cells in future work.
Another possibility is developing a (non-commutative)
3D rotation group equivariant recurrent circuits to explain the 3D rotation encoding in the brain.
These studies form our future research.
Acknowledgments
Y. W. is supported by NSF DMS-2015577.
S. W. is supported by Science and Technology Innovation 2030-Brain Science and Brain-inspired Intelligence Project (No. 2021ZD0200204) and Beijing
Academy of Artiﬁcial Intelligence.