A time-resolved theory of information encoding in recurrent neural networks
Abstract
Information transmission in neural circuits depends on how well time-varying stimuli are encoded by neural populations.
Slow neuronal timescales, noise and network chaos can compromise reliable and rapid population response to external stimuli.
A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed as a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing.
Here, we develop a non-stationary dynamic mean-ﬁeld theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding.
We demonstrate that the mutual information rate of a time-varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network ﬂuctuations.
We corroborated our ﬁndings in numerical experiments and demonstrated that recurrent networks with positive ﬁring rates trained to transmit a time-varying stimulus generically use recurrent inhibition to increase the information rate.
We also found that networks trained to transmit multiple independent time-varying signals spontaneously form multiple local inhibitory clusters, one for each input channel.
Our ﬁndings suggest that feedforward excitatory input and local recurrent inhibition–as observed in many biological circuits–is a generic circuit motif for encoding and transmitting time-varying information in recurrent neural circuits.
Introduction
How fast and reliable time-varying incoming stimuli are encoded in the population activity of recurrent neural networks constrains information transmission between local circuits and inter-areal communication.
While the brain has to respond rapidly and reliably to changes in the world, several factors can drastically hamper information encoding in recurrent network models: the timescales of synapses can be slow, single neurons exhibit Poisson-like temporally irregular dynamics and dynamic instability of recurrent neural circuit dynamics can be a source of chaotic variability [1, 2, 3, 4, 5].
Excitation and inhibition in biological neural networks is usually conveyed by different types of neurons with a predominance of recurrent inhibitory feedback, a property known as ’inhibition dominance’ [6, 7, 8, 9].
In networks with very strong recurrent inhibition, where recurrent weights are scaled as 1 / √ N with network size N , a dynamic balance arises between excitatory currents and recurrent inhibitory currents that was originally proposed as a robust mechanism to describe the emergence of asynchronous irregular activity[1].
Networks of binary neurons in such a balanced state track time-varying signals [2].
While subsequent experimental work found evidence for a dynamic balance of input currents, the ’tightness’ or ’looseness’ of such a dynamic balance in cortical circuits, as well as the computational implications, are a subject of active scientiﬁc debate [10]. ’
Loosely’ balanced networks, where excitatory and inhibitory currents are respectively O (1) compared to the distance from reset to threshold, can respond in their population ﬁring rate nonlinearly to external inputs, which was argued to be favorable for sensory processing [7]. ’
Tight balance’ refers to a more precise tracking of total excitatory and total inhibitory input currents in time originating from excitatory and inhibitory currents that are respectively O ( √ N ) or larger compared to the distance from reset to threshold [10, 1, 2].
Such ’tight balance’ was also studied in a series of works that arrived at spiking balanced networks from a normative predictive coding ansatz [11, 12, 13, 14].
How the dynamic tracking of time-dependent inputs described in binary networks [1, 2] extends to ﬁring rate models and how ’tightness of balance’ affects the information encoding of time-varying stimuli in the presence of chaos and noise has not yet been addressed.
Previous dynamic mean-ﬁeld theory (DMFT) approaches to input-driven rate networks assumed that the mean of the external inputs across neurons does not depend on time, which facilitates DMFT [15, 16, 17, 14], but prohibits to investigate encoding of dynamic stimuli in the population ﬁring rate.
To address this gap, we study how a time-varying stimulus is encoded in the population rate of an inhibition-dominated rate network under the inﬂuence of additive noise and chaos.
We show that the accuracy of neuronal population encoding improves in more tightly balanced recurrent networks because of a speedup of the effective timescale of the population mean dynamics.
Conventional methods of dynamic mean-ﬁeld theory [18, 19, 20, 17] are not adequate to capture the effects of a time-varying common input.
Therefore, we use a dynamic mean-ﬁeld theory that is non-stationary, meaning that the order parameters are time-dependent.
Beyond a similar recent approach [21], our theory can treat both time-varying common and independent external input, which is crucial to analytically treat a signal-to-noise ratio that constrains population coding.
This novel technique accurately captures the statistics of the input-driven networks.
Speciﬁcally, we calculate the cross-spectral density between input and output and the power spectral density of the population ﬁring rate.
Together with the knowledge of the input statistics, this allows us to calculate the mutual information between stimulus and population rate in the Gaussian channel approximation based on the spectral coherence.
We examine how the frequency-response and mutual information rate depends on the tightness of balance, added noise, network chaos, and statistics of the input stimulus, using both theory and simulation.
All the analytic results match those from network simulations.
We show that recurrent networks that are trained on tracking a time-dependent stimulus develop strong recurrent inhibition and strong positive input weights, a ﬁngerprint of the balanced state.
Concomitantly, the mutual information rate between their network readout and the stimulus increases, as predicted by our theory.
This indicates that a more tightly balanced state is a generic solution to reliably transmitting a time-varying input in the presence of noise or chaos.
Lastly, we ﬁnd that networks trained on simultaneously transmitting multiple independent time-varying stimuli spontaneously break up into weakly connected subnetworks with strong local inhibition.
Our ﬁndings have important implications for information encoding in RNNs and for understanding how neural network architecture design shapes noise-robustness and information encoding.
The main contributions are the following:
• A novel non-stationary mean-ﬁeld theory describing the dynamics of balanced RNNs driven by an time-varying input signal and noise (section 2 and appendix A).
• A transparent mathematical link between tightness of balance and the mutual information rate between a scalar input signal and the population rate (section 4 and appendix D).
• A frequency-resolved mutual information rate analysis showing the effects of noise and chaos (section 3 and appendix C).
• Training results indicating that vanilla networks trained on transmitting high-frequency signals become more tightly balanced throughout training (section 6 and appendix E).
• Additional training results showing that networks trained to transmit multiple independent signal exhibit a spontaneous symmetry breaking into weakly connected subnetworks with strong local inhibition (section 7 and appendix I).
• Additional training results with a nonlinear task (section 8), different activation function (appendix H) and excitatory
-inhibitory architecture (G) demonstrate generality of our results.
Population coding in recurrent networks
We study how well a time-varying scalar input I ( t ) is encoded in the mean population ﬁring rate ν ( t ) = 1 N (cid:80) i φ ( h i ( t )) of a recurrent network of N nonlinear rate units (’neurons’) that obey τ d h i d t = − h
i +
N (cid:88) j =
J ij φ ( h j ) + bI ( t ) + ξ i ( t ) , (1) with the coupling weights J ij = − bJ 0 /N
+ ˜ J ij .
The ˜ J ij are drawn independently from a Gaussian distribution with zero mean and variance g 2 /N , where g is a gain parameter that controls the heterogeneity of weights.
We use a threshold-linear transfer function φ ( x ) = max( x, 0) .
The
[Figure]
E theory simulations
[Figure]
external input contains a signal component
I ( t ) , which is identical across neurons, and a noise term ξ i ( t ) , which is independent across neurons.
For concreteness, we choose ξ i to be independent additive white Gaussian noise processes (AWGN) with autocorrelation function (cid:104)
ξ i ( t ) ξ i ( t + t (cid:48) )
(cid:105) =
τσ 2 δ ( t (cid:48) ) and the signal I ( t ) to be an Ornstein-Uhlenbeck process given by
[Equation]
The amplitude of the AWGN ξ S ( t ) driving the time-varying part is denoted by I 1 .
The parameter b multiplies both external input and recurrent coupling strength and thus regulates the tightness of balance [14, 10, 21].
For ﬁring-rate networks, the ability to track time-dependent input strongly depends on the strength of balance b (Fig 1).
For loosely balanced network ( b = 1 ) (Fig 1B), the network population rate only tracks slow ﬂuctuations of the input signal.
In contrast, for tightly balance networks (Fig 1C), the population ﬁring rate also tracks fast input ﬂuctuations.
To understand how this tracking arises in the model, it is useful to rewrite Eq 1 by decomposing h i
= m
+ ˜ h
i into a common component and residual ﬂuctuations.
With J ij = − bJ 0 /N
+ ˜
J ij , this results in
[Equation]
Here, the signal I ( t ) enters the expression for m , because it is identical across all neurons.
It thus directly impacts the mean population rate ν ( t ) and recruits, through the negative mean coupling − bJ 0 , strong recurrent feedback − bJ
ν ( t ) that is anti-correlated with the input and cancels most of the common external input
I ( t ) .
Rewriting Eq 3a by solving for ν ( t ) gives
[Equation]
This equation is commonly referred to as the ’balance equation’ [2, 19, 20] when the input is not time-dependent.
For large b , the population rate approaches ν ( t ) =
I ( t ) /J 0 , which indicates a linear relationship between input and population rate for tightly balanced networks as pointed out before in binary networks [1, 2].
On the other hand, the noise ξ i affects only the residual ﬂuctuations ˜ h
i .
As the balance parameter b is increased, the effective timescale of the dynamics of m becomes shorter by a factor of b , as can be seen by dividing Eq 3a by b resulting in a timescale of τ/b .
This leads to faster and more precise tracking.
In contrast, the timescale of the equation for ˜ h
i is unchanged by b .
Assuming a separation of timescales, for increasing b , the linearized rate dynamics ν develops a negative real outlier eigenvalue of size − b .
Of course, the two equations for m and ˜ h
i are interdependent via φ ( h i ) =
φ ( m + ˜ h i ) , so for a complete treatment, the joint dynamics of m and ˜ h i has to be solved.
In appendix A we present a non-stationary dynamic mean-ﬁeld theory of temporal population coding.
Frequency-dependent signal encoding in the population response
We systematically analyze how different input frequencies are encoded in the population response and observe that the transmission of high frequencies stimuli is improved for large b (’tightly balanced’ regime)
(Figure 2A and C).
We calculate the dynamic gain G ( f ) =
| S Iν | / | S II | of the recurrent network both in direct numerical simulations and with the non-stationary DMFT.
S Iν is the Fourier transformation of the input-output cross-correlation function between I ( t ) and ν ( t ) .
S II is the power spectral density of the stimulus I ( t ) .
The dynamic gain quantiﬁes how a variation in the input signal I ( t ) affects the population rate ν ( t ) and is a standard analysis for characterizing the response properties of individual cells [22].
We calculate G ( f ) both using both direct numerical network simulations and dynamic mean-ﬁeld theory.
(For details of the spectral analysis, see appendix C.)
We observe that increasing b boosts the dynamic gain for high frequencies (Figure 2A and C for g =
and g = 2 ).
This is a direct consequence of the speedup of the dynamics of the mean discussed above.
Thus, as the network becomes more tightly balanced, high-frequency signals can be encoded and transmitted more reliably.
To quantify the improved information encoding we turn to an information-theoretic analysis next.
Mutual information rate between time-varying stimulus and population response
With information theory, we can treat the recurrent neural network as a noisy communication channel, transforming a signal embedded in a noisy current into a population rate that can be read out by another population. We use a Gaussian channel approximation to the mutual information rate between
[Figure]
y n a m i
c g a i n ( 1 / )
[Figure]
u t u a l i n f
[Figure]
r m a t
i
o
n d e n s i t y
( b i t / (
[Figure] [Figure]
i n f o
[Figure]
m
[Figure]
t i o n d e n s i t y
( b i t / ( 1 / )
DDD
, g = 0 for A) and B) .
g = 2 for C) and D) , ∆ t = 1 / 2 10 τ , t sim = 2 6 τ , I
= J
= 1 , τ S = 1 .
τ , σ = 0 .
1 , I 1 = 0 .
input current I ( t ) and output mean population rate ν ( t ) based on the spectral coherence [23, 24]: [Equation]
where C Iν ( f ) denotes the magnitude squared spectral coherence.
The spectral coherence is the frequency-domain analog of the correlation and measures the linear relationship between frequency components of input and output signal.
Its magnitude square is
[Equation]
It is normalized by the power spectrum S II ( f ) of the input signal (an OU process) and the power spectrum S νν ( f ) of the population rate ν ( t ) .
We calculated the spectral coherence in two independent ways: ﬁrstly based on the non-stationary
DMFT and secondly based on direct numerical simulations.
The dependence of C Iν ( f ) on the tightness of balance b is depicted in Figure 1D. DMFT and direct numerical simulations with additive ted white noise ξ ( t ) are in excellent agreement.
We ﬁnd that the mutual information rate grows approximately linear with b for and OU-input signal and Gaussian white noise (see Figure 1E and appendix D).
This is consistent with an analytical derivation of the mutual information rate in a linear approximation, where we ﬁnd the mutual information rate (in nats/τ ) to be
[Equation]
for bI 1 (cid:112)
τ S /τ
(cid:29) σ/ √ N , i.e., for large networks and large signal-to-noise ratio.
The mutual information rate saturates for large b ; the saturation level is determined by the band limit of the incoming signal and noise (see ﬁgure in appendix C).
We conclude that the mutual information rate scales approximately linearly with the tightness of balance b for a sufﬁciently high band limit.
Where does this linear scaling comes from?
The reason is that the b determines the cutoff frequency, up to which the network transmits information faithfully.
This conclusion is not restricted to networks of rectiﬁed linear units, a similar reasoning holds also e.g. for the threshold-power-law nonlinearities and other neuron models (See e.g. [19, 20] for balanced rate networks with other input-output transfer functions).
Finally, we note that for an input signal that is not an OU-process, we would expect a different scaling of the mutual information rate.
Frequency-resolved mutual information rate analysis
Inspired by recent work on the frequency-resolved information encoding rate in single cells [25], we analyze the population rate of RNNs and study how the encoding of different frequency components of the input signal depends on the tightness of balance b , the strength of added noise σ and the heterogeneity of the recurrent weights g .
For that, we considered the mutual information rate density r ( f ) , which gives a frequency-resolved quantiﬁcation of the information encoding rate (See for details of deﬁnition [25]):
[Equation]
Note that the integral over frequencies of r lb ( f ) gives the Gaussian channel approximation of the mutual information rate, we thus call it mutual information rate density and remark that it has units of bit/τ 1 /τ .
We observe that more tightly balanced networks can transmit more high-frequency information (Figure 2B and D for both g = 0 and g = 2 ).
Overall, the information rate for more tightly balanced networks is increased because of the boost of the signal-to-noise ratio coming from the fact that the signal I ( t ) is scaled by b , while the noise is independent of b .
The above-described frequency-dependent effect comes on top of that.
For recurrent networks with weight heterogeneity ( g > 0 ), there are additional slow residual ﬂuctuations in ˜ h
that act as an additional source of variability, which leads to a dip in the mutual information density for low frequencies (Compare Figure 2B for g = 0 and D for g = 2 ).
These low-frequency ﬂuctuations have a different effect than additive Gaussian white noise (see also [14]).
For more tightly balanced networks (larger b ), the residual ﬂuctuations do not impair information encoding (see darker green lines in ﬁgure 2B), because they are canceled.
A related phenomenon was observed in densely connected recurrent binary networks ([26, 27]).
That variance originating from weight heterogeneity ( g > 0 ) is more effectively reduced by balanced networks as a function of b compared to variance originating from added Gaussian white noise σ was already observed in rate networks with static input [14].
Training networks on auto-encoding results in tight balance
We corroborate the theoretical ﬁnding that the tightness of balance has a crucial role in information encoding in numerical experiments in trained recurrent networks, which demonstrates the generality of our ﬁndings.
For that, we train recurrent networks on an auto-encoder task (Fig 3).
We initialize small-sized recurrent networks ( N = 100 ) that follow the dynamics of Eq.
1 in a loosely balanced state, with recurrent weights drawn i.i.d.
from a Gaussian J ij ∼ N ( − 1 /N, g 2 /N )
and input weights w in i and output weights w out i are drawn from a Gaussian distribution with a positive mean.
This initialization would correspond to b = 1 in our theoretically studied model (Eq. 1).
The network receives the time-varying input signal I i ( t )
=
w in i
I ( t ) , so in contrast to our theoretical setup, in principle each neuron can receive the input signal I ( t ) with a different scale w in i .
We then trained input weights w in , output weights w out and recurrent weights J ij using backpropagation-through-time with the ADAM optimizer with standard hyperparameters (see appendix E for additional details on training setup and additional controls of training results).
We minimize the mean squared loss l = (cid:82) | ˆ I ( t ) − I ( t ) | 2 d t between a time-varying input signal I ( t ) and a linear readout of the recurrent network activity ˆ I ( t )
= 1 /N (cid:80) i w out i φ ( h i ( t )) .
Initially, the network only poorly tracks the time-varying input signal, as theoretically expected for small b .
However, we ﬁnd that the network learns to track the time-varying input signal (Figure 3A), as the test error falls throughout training epochs (Figure 3D).
Analyzing the eigenvalues of the trained network, we observe an outlier eigenvalue with a very negative real part after training that indicates an emerging strongly negative mode implementing the fast tracking (Figure 3B).
This is conﬁrmed by measuring the empirical tightness of balance ˆ b in the connectivity, which we quantify simply by the absolute value of the mean of the sum of incoming recurrent weights into each neuron ˆ b = | 1 /N (cid:80) i (cid:80) j J ij | .
This deﬁnition of ˆ b is consistent with the deﬁnition of b in our theory above.
We ﬁnd that the trained networks that are initialized with b = 1 become progressively more tightly balanced (Figure 3C) over training epochs, as can be seen by a growing ˆ b and by an improving high-frequency encoding (Figure 3E).
We prematurely stop after 100000 training epochs to avoid
Time ( ) 0.
r e a d o u t
A target pre training post training
Real( i )
I m a g i n a r y ( i )
B pre training post training 0.
Epoch 1e    b a l a n c e b
C 0.
Epoch
[Figure] [Figure]
t e s t
e rr o r
D
Frequency (1/ )
D
[Figure]
n a m i
c g a i n ( 1 / )
E 0.
Epoch 1e
M u t u a l i n f o r m a t
i o n ( b i t / )
F pre training post training  1e   become more numerical instabilities for very tightly balanced networks.
We note that the average of the external input weights 1 N (cid:80) i w in i becomes more positive over training epochs (Figure 3C inset, green line) while the average row sum of the recurrent weights (Figure 3C inset, black line) become more negative.
Consistent with our theory, we ﬁnd that the mutual information rate (Gaussian channel approximations) grows over training epochs as the network becomes more tightly balanced (Fig 3F).
Note that we did not train the network to maximize the mutual information rate, but to minimize the mean-squared error between target output and actual output, but the two are closely related [28].
We note that this tracking does not rely on the threshold-linear input-output function, in fact, we also successfully trained sigmoid (Fig. 9), linear (Fig. 10) and threshold-quadratic input-output function φ (Fig. 11) (see appendix H for additional results).
Training on multiple inputs results in tightly balanced subnetworks
We note that networks trained to simultaneously track multiple input Ornstein-Uhlenbeck signals by a linear readout exhibit a spontaneous symmetry breaking into weakly connected subnetworks with strong local inhibition (Fig. 4 and see appendix F for results on training on multiple signals).
We note that this ﬁnding is not limited to training networks on two independent OU-processes; strongly inhibitory coupled subnetworks also emerge when training on more OU-processes (See Figure 4F for the resulting connectivity of a network trained on 3 OU-processes).
For networks trained on multiple time-varying stimuli, we observed an interesting training dynamics.
First, they develop a single global inhibitory mode (ﬁgure 7), similar to networks trained to track a single OU-signal.
After some time, we observe the emergence of a second outlier eigenvalue.
Concomitantly, the network breaks into two inhibitory subnetworks and the readouts start to track both OU-input signals independently
[Figure]
A) RNNs are trained to approximate two time-varying external inputs I i ( t ) by two linear readouts
ˆ I i ( t )
= 1 /N (cid:80) j w out ij φ (
Model parameters at initialization: N = 100 , g = √ 2 , ∆ t = 0 .
2 τ , t sim = 100 τ, b = 1 , I
= J
= 1 , τ S = 0 .
1 , σ = 1 , epochs = 10 5 .
Training on nontrivial computation results in tightly balanced network
Finally, we demonstrate the generality of our ﬁndings on two nontrivial tasks where the RNN not only copies a time-varying input but performs a computation on the input streams.
Firstly, we trained recurrent networks that receive two time-varying input signals to output a continuous extension of the XOR function f
( I 1 ( t ) , I 2 ( t ))
=
| I 1 ( t ) − I 2 ( t ) | , which satisﬁes for I 1 , I 2 ∈ 0 ,the binary XOR function (Fig 4D).
A similar, but not time-dependent task was used earlier to demonstrate that balanced networks can do nonlinear computations[29].
Consistent with our theory, we observe that the networks developed an outlier eigenvalue with strongly negative real part (Fig 4A).
K-means clustering of the recurrent weight matrix revealed that the RNNs developed tightly balanced subnetworks throughout training (Fig 4E).
See appendix F for further details.
Note that a substantial fraction of neurons in the RNN are silent after training, we leave a further analysis of these silent fraction for future work.
Secondly, we trained RNNs to compute a linear transformation of time-varying vector of stochastic processes y i ( t ) =
(cid:80)
j
A ij I j ( t ) at every moment in time.
We observed that the RNNs develop a number of negative outlier eigenvalues and concomitantly the same number of tightly balanced inhibitory subnetworks that corresponds to the rank of the linear input-output transformation A (See
Fig 12 and appendix F for further details).
Limitations
For mathematical tractability, we considered ﬁring rate networks here, but for more detailed neuron models, other biophysical features shape the information encoding rate.
For example, in spiking neuron models, ﬂuctuating background input can enhance the high-frequency encoding [30].
Moreover, the spike generation mechanism [31, 32, 33, 34, 35, 36], the synaptic dynamics ([37, 38]) and other latent variables (e.g. adaptation [39]) shape the frequency-response.
It remains an important future work to extend the work here by such contributions.
Moreover, moving beyond point-neurons, the shape of the dendritic tree can also affect the frequency response[36]. While we show that the dynamics of RNNs trained on several linear and nonlinear tasks is consistent with our DMFT, it remains an important challenge to consider tasks with more complex temporal structure, e.g. involving memory, where the task dynamics might interfere with the tightness of balance.
Finally, our theory becomes exact only in the limit of large network size, and it is important to also consider ﬁnite-size effects.
For ﬁnite network size, both noise and chaotic ﬂuctuations also contribute to ﬂuctuations of the mean m , which are also recurrently canceled.
Discussion
We show how a tight balance of excitatory currents by recurrent inhibition improves information encoding of a time-varying signal.
We demonstrate that the mutual information rate of a time-varying signal increases linearly with the tightness of balance, both in the presence of additive noise and with recurrent chaotic network ﬂuctuations.
A non-stationary dynamic mean-ﬁeld theory reveals a separation of timescale between the mean currents and the timescale of residual ﬂuctuations.
The mean dynamics become linearly faster with the tightness of balanced and enable reliable encoding of time-varying signals.
In contrast, the timescale of the chaotic dynamics in the residual ﬂuctuations are largely unaffected by the tightness of balance.
We ﬁnd that networks become more robust to deteriorating effects of ﬂuctuations from noise and chaos as the network becomes more tightly balanced.
Our study is relevant in the recent debate on the functional implications of how tightly excitatory currents are tracked by recurrent inhibition [26, 27, 7, 10, 13, 14].
We address this question by building a bridge from information-theoretic measures of information encoding that were previously used in neuroscience mostly in sensory systems [40, 41, 42, 23, 24] to dynamic mean-ﬁeld descriptions of recurrent network dynamics that were previously used to describe the often chaotic dynamics of recurrent rate networks [18, 19, 20, 16, 43, 17, 44, 14].
Besides the implications on information encoding of more loosely or more tightly balanced networks, of course also biophysical, energetic, and evolutionary constraints should be considered.
Biological networks can naturally not be arbitrarily tight balanced, which would require arbitrary large synaptic currents.
Very tight balance might also be questionable for energetic reasons, as was asked previously, "Why should the cortex simultaneously push on the accelerator and on the brake?"
However, such biophysical and energetic constraints may be better addressed in biophysically more detailed models.
We found that training networks on tracking a time-dependent signal by a linear readout by minimizing the squared error make them more tightly balanced, as reﬂected in more negative mean recurrent weights and more positive feedforward input weights.
Moreover, the training also increased the mutual information rate between the input signal and linear readout.
This is consistent with our theoretical result.
Furthermore, the fact that training arrives at a tightly balanced solution may suggest that this is a typical solution for a network.
It is an interesting question when such balanced solutions will emerge with biologically plausible learning rules [46, 47, 48, 49])
The dynamic cancellation of time-varying input through recurrent inhibitory feedback has been previously studied in balanced networks with binary [2, 26, 27] and spiking neurons [50, 51].
Chaos in balanced ﬁring-rate networks was studied previously [19, 20, 52, 17], but the dynamic cancellation of correlated input and its implications on information encoding in rate networks were not investigated, nor were the implications for training such networks in a machine learning setup with backpropagation through time.
It would be interesting to extend our mean-ﬁeld analysis to rate networks with pre-existing low-rank structures on top of the random structure [53, 54, 55, 56].
The underlying mechanisms of tracking of time-varying input we analyze here are not speciﬁc to fullyconnected threshold-linear RNNs driven by Ornstein-Uhlenbeck signals with additive Gaussian white noise, which we merely chose for the sake of simplicity and analytical tractability.
Rate networks with other non-negative input-output transfer functions exhibit a qualitatively similar dynamic tracking in the tightly balanced regime.
Moreover, the mechanism of tracking described here is tightly related to balanced predictive coding networks [13, 14].
A testable hypothesis based on our ﬁndings is that changing the tightness of balance, e.g. by pharmacological manipulation or by a genetic knockout that affects net recurrent inhibitory strength (without generating runaway activity or pathological network states), would also affect the highfrequency stimulus encoding and information transmission.
Speciﬁcally, we predict that stronger effective recurrent weights in conjunction with stronger excitatory inputs would improve both the high-frequency encoding and the mutual information rate between a stimulus and the population response.
Conversely, when weakening recurrent inhibition and external input strength, we predict an impaired high-frequency encoding and a lower mutual information rate.
Such predictions could not only be tested in vivo, but also by in vitro experiments, where the number of recurrent synapses can be manipulated [57].
Acknowledgments and Disclosure of Funding
Research supported by NSF NeuroNex Award DBI-1707398 (RE), the Gatsby Charitable Foundation (RE), the Bernstein Award 2014, 01GQ171 (SG), and the Swartz Foundation Award (2021-6) (RE).
We thank L.F. Abbott, A. Ingrosso, C. Cueva, J. Kadmon, J. Keijser, R. Khajeh, L. Logiaco, A. Neef,
K. Rajan, K. Miller, S. Solla, C. v. Vreeswijk ( † ) and F. Wolf for fruitful discussions.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] , not applicable (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] , see result section and appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplement and here training-balanced-nets (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See training section and supplement (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars for ﬁgure 1E are smaller than symbol size, ﬁgures on mutual information density and gain included a comparison to theory as check of error (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See appendix C 4.
If (a) If your work uses existing assets, did you cite the creators?
[N/A] Does not apply (b) Did you mention the license of the assets?
[N/A] Does not apply (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] Does not apply (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] Does not apply (e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] Does not apply 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] Does not apply (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] Does not apply (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Does not apply
A Non-stationary dynamic mean-ﬁeld theory with both independent and common input
The core idea of dynamic mean-ﬁeld theory is that for large N , the distribution of the residual recurrent input for different neurons becomes Gaussian and pairwise uncorrelated, according to the central limit theorem.
To this end, we characterize the distribution of the ˜ h
i ( t ) (Eq 3b) by considering the (linear) stochastic dynamics:
[Equation]
(9) where η ( t ) is a Gaussian process with mean (cid:104)
η ( t ) (cid:105) = 0 and autocorrelation
[Equation]
Here and in the following, angular brackets denote expectation values over the distribution of the stochastic process ˜ h ( t ) , which approximates population averages in the full network.
The mean-ﬁeld estimate for the mean m ( t ) therefore evolves according to Eq 3a with ν ( t ) =
(cid:104) φ ( m ( t ) + ˜ h
( t ))
(cid:105) , the mean-ﬁeld estimate of the mean population ﬁring rate.
We obtain an expression for the time evolution of the two-time autocorrelation function c ( t, s ) =
(cid:68) ˜
h ( t )˜
h ( s ) (cid:69) , which explicitly depends on two time points.
Taking the temporal derivative of c ( t, s ) with respect to s and using Eq 8, we obtain
[Equation]
where r ( t, s ) =
(cid:68) ˜
h ( t ) ( η ( s ) + ξ ( s )) (cid:69) , which we take as an auxiliary function.
Taking the temporal derivative of r ( t, s ) with respect to t and using Eq 8 again, we arrive at
[Equation]
12) where q ( t, s )
=
g 2 (cid:10) φ ( m ( t ) + ˜ h [Equation] ) φ ( m ( s ) + ˜ h
( s ))
(cid:11) (see Eq 9) and σ 2 is the variance of ﬂuctuations driven by the independent noise input.
The additional term proportional to δ ( t − s ) in Eq.
11 is due to this independent additive Gaussian white noise, similarly to the independent input considered previously [17, 16].
However, we consider simultaneously a time-varying common input signal I ( t ) , which renders the problem non-stationary.
Such a non-stationary DMFT was recently developed [21], but without the independent input, which is crucial for meaningfully discussing information encoding rates in terms of a signal-to-noise ratio.
Together, the dynamic mean-ﬁeld equations for m ( t ) , c ( t, s ) and r ( t, s ) form a closed system of self-consistent dynamic equations and can be solved forward in time s and t by integrating them on a two-dimensional grid from some initial condition for m , c and r .
The integration requires q ( t, s ) , which can be calculated by evaluating a Gaussian double integral that depends on c ( t, s ) , c ( t, t ) , c ( s, s ) , m ( t ) and m ( s ) .
For the threshold-linear transfer function φ ( x ) = max( x, 0) , one integral can be evaluated analytically, which allows for an efﬁcient numerical implementation using adaptive Gauss–Kronrod integration.
The non-stationary dynamic mean-ﬁeld theory captures accurately the time-dependent mean population rate ν ( t ) and the two-time autocorrelation function from numerical simulations.
In contrast to previous work that did not include noise [21], we consider here a network where neurons simultaneously receive a time-varying input component
I ( t ) ("signal") that is identical across neurons and an input component ξ i ("noise") that is independent across neurons.
This setup is necessary to meaningfully quantify an information rate for a continuous channel, which requires a signal-to-noise ratio.
Furthermore, we go beyond the previously published non-stationary dynamic mean-ﬁeld theory [21] by introducing a heuristic term that emulates ﬂuctuating input into the m -equation due to ﬁnite network size N similar to [14] (see appendix B).
We note that in contrast to [14], we do solve the self-consistent set of dynamic mean-ﬁeld equations for m , c and r to calculate dynamic gain, mutual information rate and mutual information rate density.
B Heuristic ﬁnite-size ﬂuctuations
The non-stationary dynamic mean-ﬁeld theory is exact in the limit of large N .
However, for the precise evaluation of the spectral coherence, ﬁnite-size network corrections have to be taken into account.
These corrections are expected to be of order 1 √ N and therefore have only a marginal effect on the dynamic gain.
The ﬁnite-size ﬂuctuations can be heuristic accounted for by adding an additional ﬂuctuation term to the equation of m ( t ) (mean-ﬁeld version of Eq. 3a), analogously to previous work [14]:
[Equation]
(13) where the mean-ﬁeld estimate of the mean population ﬁring rate ν ( t ) = (cid:104) φ [Equation] m [Equation]
+ ˜ h
[Equation]
t ))
(cid:105) is additionally driven by the ﬂuctuations ζ ( t ) , a noise term that captures the effective ﬁnite-size ﬂuctuations.
In the case of g = 0 , ζ ( t ) reduces to an additive white Gaussian noise processes (AWGN) with autocorrelation function (cid:104) ζ ( t ) ζ ( t + t (cid:48) )
(cid:105) =
τσ 2 δ ( t (cid:48) ) originating from the mean across neurons of the independent input noise ξ i ( t ) .
For g > 0 , the ﬁnite size noise ζ ( t ) should include the effect coming from the residual ﬂuctuations ˜ h
i ( t ) , which adds a colored noise component to it (see also [14]).
In contrast to [14], we heuristically approximate both amplitude and timescale of the ﬁnite-size ﬂuctuations from the solution of the stationary DMFT.
C Details on numerical and analytical spectral analysis
We numerically solve the stochastic differential equation Eq.
1 using the Euler–Maruyama method but with a distributionally-exact solution of the Ornstein-Uhlenbeck process for I ( t ) , meaning that I ( t ) is not a solution of stochastic differential equations and instead is directly generated according to its analytical distributions.
To calculate the spectral coherence, we used the function coherence from the Python package scipy.signal version 1 . 7 .
3 with a default Hann window window and default parameters except that we set the length of each segment (nperseg) to 2 11 .
We calculated the integral over the mutual information density 1 − C Iν ( f ) using standard trapezoidal integration.
As pointed out in the main text, we ﬁnd that the mutual information rate approximately grows linear with tightness of balance b .
In ﬁgure 5, we demonstrate that this linear scaling over several orders of magnitude is also true for different levels of additive Gaussian white noise σ .
We note that for low noise σ and large b , the mutual information rate saturates which is an effect of ﬁnite time-resolution of the integration of both dynamic mean-ﬁeld theory and direct numerical simulations that result in an effective upper bound on the frequencies (see ﬁgure 5), which leads to a deviation from the linear scaling of the mutual information rate.
We conﬁrm this by deliberately introducing an upper cutoff frequency f cutoff up to which the spectral coherence is integrated:
[Equation]
As expected, changing this frequency cutoff directly affects the mutual information rate for large b (see ﬁgure 6).
For high noise σ and small b , we also observe deviation from the linear scaling.
In that case, for ﬁnite simulation time t sim signal is so weak that spurious correlations between input and output give the illusion of mutual information.
This lower saturation can be ameliorated by a longer simulation time.
All simulations were performed on a laptop and took minutes to hours.
D Details on analytical approximation of mutual information rate
We observed an approximately linear scaling of the mutual information rate with tightness of balance b , both for noise-driven homogeneous networks ( g = 0 ) and chaotic( g = 0 ) networks ( g > √ 2 ) that receive an OU-process as input.
Where does this linear scaling originate from?
The linear scaling of the mutual information rate can be obtained from the boost of the effective timescale of the network in combination with the signal-to-noise ratio that comes from the OU-input signal and the additive Gaussian white noise.
The boost of the effective timescale was already noted earlier [2, 14], but we provided a non-stationary mean-ﬁeld theory for nonlinear rate networks and measured the effect on the mutual information rate.
The power spectral density of the input signal is given
[Figure]
M u t u a l i n f o r m a t i o n ( b i t / ) noise strength 0.
[Equation]
[Equation]
[Equation] [Equation]
by
[Equation]
S noise ( f ) is the power spectrum of band-limited Gaussian white noise:
[Equation]
In an approximation that ignores the nonlinearity of the recurrent network dynamics and the interplay between m and ˜ h , we can approximate that the b changes the effective low-pass ﬁltering of the network from the low-pass ﬁltering coming from the leak-term in the dynamics
[Equation]
(17) to an effective ﬁlter of
[Equation]
Thus, the characteristic timescales of the mean m ( t ) and thus of the population ﬁring rate ν ( t ) is τ/b .
This results from Eq.
3, when assuming a separation of timescales between the time-varying statistics of m and ˜ h .
Thus, in a linear approximation, the magnitude square coherence can be approximated to be
[Equation]
Using the equation for the mutual information rate in the Gaussian channel approximation, this results in a mutual information density of
[Equation]
[Figure]
a l i n f o r m a t
i o n ( b i t / )
[Figure] ∆ =
2 12 τ t sim = 2 6 τ
0 =
J 0 = τ = 0 . Thus, the effective signal-to-noise ratio of a scalar signal encoded in the population rate is boosted by a factor Nb 2 .
Calculating the mutual information gives
[Equation]
(21) for bI 1 (cid:112) τ S /τ
(cid:29) σ/ √ N , i.e., for large networks and large signal-to-noise ratio, this results in a linear scaling of the mutual information rate with tightness of balance:
[Equation]
We observe already for moderate tightness of balance and moderate network size very good agreement between this linear approximation and numerical experiments.
E Details on training setup and additional numerical training experiments
An example implementation in Flux, a machine learning library in Julia is available at training-balanced-nets.
Brieﬂy, we trained standard rate networks ( Eq. 1) to track an OU-process by a linear readout.
We implemented the stochastic dynamics using the Euler–Maruyama method.
We used the default ADAM hyperparameters without any additional ﬁne-tuning and found our results to be robust with respect to choices of parameters.
The parameters of the network simulations are reported in the respective ﬁgure captions.
F Details on training networks to track multiple time-varying signals
We found that training recurrent neural networks simultaneously on multiple signals yields balanced subnetworks.
We investigated this in the following scenario:
We trained RNNs to approximate two time-varying external inputs I i ( t ) by two linear readout ˆ I ( t )
= 1 /N (cid:80) i w out i φ ( h i ) , by minimizing the mean squared loss l = (cid:82)
| ˆ I ( t ) − I i ( t ) | 2 d t .
The training scenario is identical to training on tracking a single OU-process, the only difference is that there exist two independent OU-input processes
I i ( t ) and two readout vectors w out i .
All parameters (recurrent weight matrix J ij , input weights and output weights) are initialized with parameters drawn independently identically from Gaussian distributions as before.
We found that such randomly initialized networks trained on two OU-processes spontaneously break into two balanced subnetworks throughout training.
The subnetworks have strong local inhibition among neurons belonging to one subnetwork, but weak connectivity between the subnetworks.
We displayed that ﬁnding in Figure 4.
We ﬁnd the two subnetworks by k-means clustering on the input weights.
Note that in the case of two
OU-processes, the sorting can also be obtained by simply sorting the weight matrix J ij according to the indices of the sorted input weight vectors, i.e. by performing ’sortperm’ on one of the input vectors using the resulting indices as row- and column indices of the displayed weight matrix J ij .
Furthermore, we found that the entries of the input weights, which were independent Gaussian random initially also displayed symmetry breaking.
We found that the entries of input and output weights associated with the same signal are strongly correlated (Figure 4D), but entries associated with different signals are anti-correlated (Figure 4E).
We note that this ﬁnding is not limited to training networks on two independent OU-processes; strongly inhibitory coupled subnetworks also emerge when training on more OU-processes (See Figure 4F for the resulting connectivity of a network trained on 3 OU-processes).
For networks trained on multiple time-varying stimuli, we observed an interesting training dynamics.
First, they develop a single global inhibitory mode (ﬁgure 7), similar to networks trained to track a single OU-signal.
After some intermediate time, we observe the emergence of a second negative outlier eigenvalue.
Concomitantly, the network breaks into two inhibitory subnetworks and the readouts start to track both OU-input signals independently.
An analytical analysis of this phenomenon is beyond this publication, but it can be explained in terms of the network to learn subsequent singular values of the input-output cross-correlation matrix [58], where the singular value that captures most of the variance is learned ﬁrst.
Indeed, tracking the average of the two OU-signals at each moment in time is favorable compared to e.g. tracking only one and ignoring the other.
This can be seen by calculating the expectation of the mean squared error for the two cases.
In the case where the mean is perfectly tracked, the mean squared error is by a factor of two larger in case only one is tracked and the other is ignored.
[Figure]
two readout I by minimizing the squared loss
I
) |
) When trained on two independent stimuli, two outlier eigenvalues with negative real parts emerge
) When observing the mean connectivity strength within the two subnetworks and between and the global mean coupling, and interesting training dynamics emerges: First, all weights become more negative and one single negative outlier eigenvalue evolves, then at some intermediate state, the network breaks up into two subnetworks, and their respective means become more negative but the weights between them on average actually become closer to zero.)
Model parameters at [-initialization: N = 100 , g = √ 2 , ∆ t = 0 . 2 τ , t sim = 100 τ, b = 1 , I 
 = J 
 = 1 , τ S = 0 .-]initialization
G Mixed excitatory-inhibitory circuits
To corroborate our training results, we also initialized the trained network with separate excitatory and inhibitory populations using the classical parametrization of [1], which for large b results in a balanced state.
Consistent with [1] [Figure] we chose mean couplings J EE =
J IE = 1 .
/N
, J II = − 1 . 8 /N , J EI = − 2 .
/N .
We found that training such a network on the auto-encoder task also results in a single inhibition-dominated population (See
Figure 7).
We did not try to constrain the weight update to maintain a separation of excitation and inhibition.
This interesting scenario is left for future work
[Figure]
H Training networks with other nonlinearities
To corroborate our training results and show the generality of our ﬁndings, we trained networks on the autoencod- ing task also using sigmoid (Figure 9), linear (Figure 10) and threshold-quadratic (Figure 11) activation function φ ( x ) .
We found that training such networks on the auto-encoder task also results in a single inhibition-dominated population (See Figure 9 - 11) that becomes progressively more tightly balanced over time, as seen from the outlier eigenvalue that emerges over time.
I Training on linear transformation on multiple input streams results in tightly balanced subnetworks
To furhter corroborate our training results and show the generality of our ﬁndings, we also trained networks on calculating a linear transformation on multiple input streams in each moment in time.
[Figure] a l ( i ) [Figure] For that, analogous to Fig.4B-D, RNNs are are trained to perform a linear transformation y i ( t ) =
(cid:80)
j
A ij I j ( t ) on a set of multi-dimensional time-varying external input
I i ( t ) (Fig 12 top middle)).
The training was done by minimizing the mean squared loss l
= (cid:80) j (cid:82) | ˆ y ( t ) − (cid:80) ij A ij I j ( t )
| 2 d t , where ˆ y i are linear readouts of the recurrent network, ˆ y i ( t ) = 1 /N (cid:80)
j w out ij φ ( h j ) and A is a random matrix of size S × R with entries drawn independently from a Gaussian distribution with zero mean and variance 1 /R .
Here wo chose R = S = 3 .
Again, the time-varying input signals I j ( t ) were independent OU processes.
We observe that the test loss drops throughout training (Fig 12 top left)), concomitantly the linearized network dynamics develops three outlier with negative real part throughout the course of the training (Fig 12 top right)).
We observe that the number of outlier eigenvalues is determined by the rank of A (Fig 12 bottom left)) and their training dynamics seems to be shaped by the singular values of A , reminiscent of the training dynamics of multi-layer perceptrons [58].
For our case of R = S = 3 in ﬁgure 12 we observed that three outlier eigenvalues develop, at the same time three tightly balanced subnetworks with strong inhibition emerge (Fig 12 bottom right)).
For illustration, we plotted the activity
h
i for 5 example neurons in the bottom middle of Fig 12.
Consistent with our theory, the network becomes more tightly balanced over the course of training.
We leave an analytical description of the training dynamics for future work.
[Figure] [Figure]
Epoch 1e 101010t
e s t
e rr o r
Time ( )   inputtargetsoutput
Time ( )
e x a m p l e h  real( i ) 0.
i m a g ( i ) initialtrained
Epoch 1e
r e a l ( i )     50   0.
[Table]