A time-resolved theory of information encoding in recurrent neural networks
Anonymous Author(s)
AfﬁliationAddress
Abstract
Mammalian brains process information by the collective dynamics of a deeplylayered structure of recurrent networks.
Information transmission in neural circuitsdepends on how well time-varying stimuli are encoded by neural populations.
Adynamic balance of externally incoming currents by strong recurrent inhibitionwas previously proposed as a mechanism to accurately and robustly encode theinformation in a time-varying stimulus, but a full theory was missing.
Here, wedevelop a non-stationary dynamic mean-ﬁeld theory that transparently explains howa tight balance of excitatory currents by recurrent inhibition improves informationencoding.
We demonstrate that the mutual information rate of a time-varying inputincreases linearly with the tightness of balance, both in the presence of additivenoise and with recurrently generated chaotic network ﬂuctuations.
We corroboratedour ﬁndings in numerical experiments and demonstrated that recurrent networkswith positive ﬁring rates trained to transmit a time-varying stimulus generically userecurrent inhibition to increase the information rate.
We also found that networkstrained to transmit multiple independent time-varying signals spontaneously formmultiple local inhibitory clusters, one for each input channel.
Our ﬁndings suggestthat feedforward excitatory input and local recurrent inhibition–as observed inmany biological circuits–is a generic circuit motif for encoding and transmittingtime-varying information in recurrent neural circuits.
Introduction
How fast and reliable time-varying incoming stimuli are encoded in the population activity ofrecurrent neural networks constrains information transmission between local circuits and inter-arealcommunication.
While the brain has to respond rapidly and reliably to changes in the world, thetimescales of synapses can be slow, single neurons exhibit Poisson-like temporally irregular dynamicsand dynamic instability of recurrent neural circuit dynamics can be a source of chaotic variability[1, 2, 3, 4, 5].
Excitation and inhibition in biological neural networks is usually conveyed by different types ofneurons with a predominance of recurrent inhibitory feedback, a property known as ’inhibitiondominance’ [6, 7, 8, 9].
In networks with very strong recurrent inhibition, where recurrent weightsare scaled as 1 / √ N with network size N ), a dynamic balance arises between excitatory currentsand recurrent inhibitory currents that was originally proposed as a robust mechanism to describe theemergence of asynchronous irregular activity[1].
Networks of binary neurons in such a balanced statetrack time-varying signals [2].
While subsequent experimental work found evidence for a dynamicbalance of input currents, the ’tightness’ or ’looseness’ of such a dynamic balance in cortical circuits,as well as the computational implications, are a question of active scientiﬁc debate [10]. ’
Loosely’balanced networks, where excitatory and inhibitory currents are respectively O (1) compared to the distance from reset to threshold, can respond in their population ﬁring rate nonlinearly to externalinputs, which was argued to be favorable for sensory processing [7].’Tight balance’ refers to amore precise tracking of total excitatory and total inhibitory input currents in time originating fromexcitatory and inhibitory currents that are respectively O (cid:16) √ N (cid:17) or larger compared to the distancefrom reset to threshold [10, 1, 2].
Such ’tight balance’ was also studied in a series of works thatarrived at spiking balanced networks from a normative predictive coding ansatz [11, 12, 13, 14].
It is important to investigate how such biological features such as ’tightness of balance’ shape thereliability of population response to external stimuli.
How the dynamic tracking of time-dependentinputs described in binary networks [1, 2] extends to ﬁring rate models and how it affects theinformation encoding of time-varying stimuli in the presence of chaos and noise has not yet beenaddressed.
Previous dynamic mean-ﬁeld theory (DMFT) approaches to input-driven rate networksassumed that the mean of the external inputs across neurons does not depend on time, which facilitates
DMFT [15, 16, 17, 14], but prohibits to investigate encoding of dynamic stimuli in the populationﬁring rate.
To address this gap, we study how a time-varying stimulus is encoded in the populationrate of an inhibition-dominated rate network under the inﬂuence of additive noise and chaos.
We show that the accuracy of neuronal population encoding improves in more tightly balancedrecurrent networks because of a speedup of the effective timescale of the population mean dynamics.
Conventional methods of dynamic mean-ﬁeld theory [18, 19, 20, 17] are not adequate to capturethe effects of a time-varying common input.
Therefore, we use a dynamic mean-ﬁeld theory thatis non-stationary, meaning that the order parameters are time-dependent.
Beyond a similar recentapproach [21], our theory can treat both time-varying common and independent external input, whichis crucial to analytically treat a signal-to-noise ratio that constrains population coding.
This noveltechnique accurately captures the statistics of the input-driven networks.
Speciﬁcally, we calculatethe cross-spectral density between input and output and the power spectral density of the populationﬁring rate.
Together with the knowledge of the input statistics, this allows us to calculate the mutualinformation between stimulus and population rate in the Gaussian channel approximation based onthe spectral coherence.
We examine how the frequency-response and mutual information rate depends on the tightness ofbalance, added noise, network chaos, and statistics of the input stimulus, using both theory andsimulation.
All the analytic results match those from network simulations.
We show that recurrentnetworks that are trained on tracking a time-dependent stimulus develop strong recurrent inhibitionand strong positive input weights, a ﬁngerprint of the balanced state.
Concomitantly, the mutualinformation rate between their network readout and the stimulus increases, as predicted by ourtheory.
This indicates that a more tightly balanced state is a generic solution to reliably transmittinga time-varying input in the presence of noise or chaos.
Lastly, we ﬁnd that networks trained onsimultaneously transmitting multiple independent time-varying stimuli spontaneously break up intoweakly connected subnetworks with strong local inhibition.
Our ﬁndings have important implications for information encoding in RNNs and for understandinghow neural network architecture design shapes noise-robustness and information encoding.
The main contributions are the following:
• A novel non-stationary mean-ﬁeld theory describing the dynamics of balanced RNNs driven
by an time-varying input signal and noise (section 2 and appendix A).
• A transparent mathematical link between tightness of balance and the mutual information rate between a scalar input signal and the population rate (section 4 and appendix D).
• A frequency-resolved mutual information rate analysis showing the effects of noise andchaos (section 3 and appendix C).• Numerical network simulations showing an excellent agreement with the theoretical results
• Training results indicating that vanilla networks trained on transmitting high-frequencysignals become more tightly balanced throughout training (section 5 and appendix E).• Additional training results showing that networks trained to transmit multiple independentsignal exhibit a spontaneous symmetry breaking into weakly connected subnetworks withstrong local inhibition (section 6 and appendix I).
Additional training results with a nonlinear task (section 7), different activation function(appendix H) and excitatory-inhibitory architecture (G) demonstrate generality of our results
Population coding in recurrent networks
We study how well a time-varying scalar input I ( t ) is encoded in the mean population ﬁring rateν ( t ) = 1 N (cid:80) i φ ( h i ( t )) of a recurrent network of N nonlinear rate units (’neurons’) that obeyτ d h i d t = − h
i +
N (cid:88) j =
J ij φ ( h j ) + bI ( t ) + ξ i ( t ) , (1) with the coupling weights J ij = − bJ 0 /N
+ ˜ J ij .
The ˜ J ij are drawn independently from a Gaussiandistribution with zero mean and variance g 2 /N , where g is a gain parameter that controls theheterogeneity of weights.
We use a threshold-linear transfer function φ ( x ) = max( x, 0) .
The
[Figure]
E theory simulations
[Figure]
P o p u l a t i o n r a t e C
C
96 external input contains a signal component I ( t ) , which is identical across neurons, and a noise termξ i ( t ) , which is independent across neurons.
For concreteness, we choose ξ i to be independent additivewhite Gaussian noise processes (AWGN) with autocorrelation function (cid:104)
ξ i ( t ) ξ i ( t + t (cid:48) )
(cid:105) =
τσ 2 δ ( t (cid:48) )and the signal I ( t ) to be an Ornstein-Uhlenbeck process given by
[Equation]
The amplitude of the AWGN ξ S ( t ) driving the time-varying part is denoted by I 1 .
The parameterb multiplies both external input and recurrent coupling strength and thus regulates the tightness ofbalance [14, 10, 21].
For ﬁring-rate networks, the ability to track time-dependent input strongly depends on the strength of balance b (Fig 1).
For loosely balanced network ( b = 1 ) (Fig 1B), thenetwork population rate only tracks slow ﬂuctuations of the input signal.
In contrast, for tightlybalance networks (Fig 1C), the population ﬁring rate also tracks fast input ﬂuctuations.
To understandhow this tracking arises in the model, it is useful to rewrite Eq 1 by decomposing h i
= m
+ ˜ h
i into acommon component and residual ﬂuctuations.
With J ij = − bJ 0 /N
+ ˜
J ij , this results in
[Equation]
Here, the signal I ( t ) enters the expression for m , because it is identical across all neurons.
It thusdirectly impacts the mean population rate ν ( t ) and recruits, through the negative mean coupling− bJ 0 , strong recurrent feedback − bJ
ν ( t ) that is anti-correlated with the input and cancels most ofthe common external input I ( t ) .
Rewriting Eq 3a by solving for ν ( t ) gives
[Equation]
This equation is commonly referred to as the ’balance equation’ [2, 19, 20] when the input is nottime-dependent.
For large b , the population rate approaches ν ( t ) =
I ( t ) /J 0 , which indicates a linearrelationship between input and population rate for tightly balanced networks as pointed out before inbinary networks [1, 2].
On the other hand, the noise ξ i affects only the residual ﬂuctuations ˜ h
i .
As the balance parameter b is increased, the effective timescale of the dynamics of m becomes τ/b ,as can be seen by dividing Eq 3a by b .
This leads to faster and more precise tracking.
In contrast, thetimescale of the equation for ˜ h
i is unchanged by b .
Of course, the two equations for m and ˜ h
i areinterdependent via φ ( h i ) =
φ ( m + ˜ h i ) , so for a complete treatment, the joint dynamics of m and ˜ h ihas to be solved.
In appendix A we present a non-stationary dynamic mean-ﬁeld theory of temporalpopulation coding.
Frequency-dependent signal encoding in the population response
[Figure]
∆ t
2 10 τ t sim = 2 6 τ
I
= J 0 =
τ S = 1 .
τ σ = 0 .
I 1 = 0 .
We systematically analyze how different input frequencies are encoded in the population response andobserve that the transmission of high frequencies stimuli is improved for large b (’tightly balanced’regime) (Figure 2A and C).
We calculate the dynamic gain G ( f ) =
| S Iν | / | S II | of the recurrentnetwork both in direct numerical simulations and with the non-stationary DMFT.
S Iν is the Fouriertransformation of the input-output cross-correlation function between I ( t ) and ν ( t ) .
S II is the powerspectral density of the stimulus I ( t ) .
The dynamic gain quantiﬁes how in linear response a variationin the input signal I ( t ) affects the population rate ν ( t ) and is a standard analysis for characterizingthe response properties of individual cells [22].
We calculate G ( f ) both using both direct numerical network simulations and dynamic mean-ﬁeldtheory.
(For details of the spectral analysis, see appendix C.)
We observe that increasing b booststhe dynamic gain for high frequencies (Figure 2A and C for g = 0 and g = 2 ).
This is a directconsequence of the speedup of the dynamics of the mean discussed above.
Thus, as the networkbecomes more tightly balanced, high-frequency signals can be encoded and transmitted more reliably.
To quantify the improved information encoding we turn to an information-theoretic analysis next.
Mutual information rate between time-varying stimulus and populationresponse
With information theory, we can treat the recurrent neural network as a noisy communication channel,transforming a signal embedded in a noisy current into a population rate that can be read out byanother population.
We use a Gaussian channel approximation to the mutual information ratebetween input current
I ( t ) and output mean population rate ν ( t ) based on the spectral coherence[23, 24]: R lb
( I, ν ) = − (cid:82) f cutoffdf log 2 (1 − C Iν ( f )) , where C Iν ( f ) denotes the magnitude squaredspectral coherence.
The spectral coherence is the frequency-domain analog of the correlation andmeasures the linear relationship between frequency components of input and output signal.
Itsmagnitude square is C Iν ( f ) =
| S Iν
( f ) | 2 S II ( f ) S νν ( f ) .
It is normalized by the power spectrum S II ( f ) of theinput signal (an OU process) and the power spectrum S νν ( f ) of the population rate ν ( t ) .
We calculated the spectral coherence in two independent ways: ﬁrstly based on the non-stationary
DMFT and secondly based on direct numerical simulations.
The dependence of C Iν ( f ) on thetightness of balance b is depicted in Figure 1D. DMFT and direct numerical simulations with additiveband-limited white noise ξ ( t ) are in excellent agreement.
We ﬁnd that the mutual information rate grows approximately linear with b (see Figure 1E).
It saturatesfor large b ; the saturation level is determined by the band limit of the incoming signal (see ﬁgure inappendix C).
We use the high-frequency behavior and low-frequency limit to derive an analyticalapproximation of the linear scaling (see appendix D).
We conclude that the mutual information ratescales approximately linearly with the tightness of balance b for a sufﬁciently high band limit.
Wheredoes this linear scaling comes from?
The reason is that the b determines the cutoff frequency, upto which the network transmits information faithfully.
Contributions from frequencies beyond thiscutoff are negligible to ﬁrst order.
This conclusion is not restricted to networks of rectiﬁed linearunits, a similar reasoning holds also e.g. for the threshold-power-law nonlinearities and other modelswith non-negative ﬁring rates (See e.g. [19, 20] for balanced rate networks with other input-outputtransfer functions).
Inspired by recent work on the frequency-resolved information encoding rate in single cells [25], weanalyze the population rate of RNNs and study how the encoding of different frequency componentsof the input signal depends on the tightness of balance b , the strength of added noise σ and theheterogeneity of the recurrent weights g .
For that, we considered the mutual information rate density r ( f ) , which gives a frequency-resolvedquantiﬁcation of the information encoding rate (See for details of deﬁnition [25]):
[Equation]
We observe that more tightly balanced networks can transmit more high-frequency information(Figure 2B and D for both g = 0 and g = 2 ).
Overall, the information rate for more tightlybalanced networks is increased because of the boost of the signal-to-noise ratio coming from thefact that the signal I ( t ) is scaled by b , while the noise is independent of b .
The above-describedfrequency-dependent effect comes on top of that.
For recurrent networks with weight heterogeneity ( g > 0 ), there are additional slow residual ﬂuctuations in ˜ h
that act as an additional source of variability, which leads to a dip in the mutual informationdensity for low frequencies (Compare Figure 2B for g = 0 and D for g = 2 ).
These low-frequencyﬂuctuations have a different effect than additive Gaussian white noise (see also [14]).
For more tightly balanced networks (larger b ), the residual ﬂuctuations do not impair informationencoding (see darker green lines in ﬁgure 2B), because they are canceled.
A related phenomenon wasobserved in densely connected recurrent binary networks ([26, 27]).
That variance originating fromweight heterogeneity ( g > 0 ) is more effectively reduced by balanced networks as a function of bcompared to variance originating from added Gaussian white noise σ was already observed in ratenetworks with static input [14].
Training networks on auto-encoding results in tight balance
[Figure]
We corroborate the theoretical ﬁnding that the tightness of balance has a crucial role in informationencoding in numerical experiments in trained recurrent networks which demonstrates the generalityof our ﬁndings.
For that, we train recurrent networks on an auto-encoder task (Fig 3).
We initializesmall-sized recurrent networks ( N = 100 ) that follow the dynamics of Eq.
1 in a loosely balancedstate, with recurrent weights drawn i.i.d.
from a Gaussian J ij ∼ N ( − 1 /N, g 2 /N ) and input weightsw in i and output weights w out i are drawn from a Gaussian distribution with a positive mean.
Thisinitialization would correspond to b = 1 in our theoretically studied model (Eq. 1).
The networkreceives the time-varying input signal I i ( t ) =
w in i
I ( t ) , so in contrast to our theoretical setup, in principle each neuron can receive the input signal I ( t ) with a different scale w in i .
We then trainedinput weights w in , output weights w out and recurrent weights J ij using backpropagation-through-timewith the ADAM optimizer with standard hyperparameters (see appendix E for additional detailson training setup and additional controls of training results).
We minimize the mean squared lossl =
(cid:82) | ˆ I ( t ) − I ( t )
| 2 d t between a time-varying input signal I ( t ) and a linear readout of the recurrentnetwork activity
ˆ I ( t )
= 1 /N (cid:80) i w out i φ ( h i ( t )) .
Initially, the network only poorly tracks the time-varying input signal, as theoretically expected forsmall b .
However, we ﬁnd that the network learns to track the time-varying input signal (Figure 3A),as the test error falls throughout training epochs (Figure 3D).
Analyzing the eigenvalues of thetrained network, we observe an outlier eigenvalue with a very negative real part after training thatindicates an emerging strongly negative mode implementing the fast tracking (Figure 3B).
This isconﬁrmed by measuring the empirical tightness of balance ˆ b in the connectivity, which we quantifysimply by the absolute value of the mean of the sum of incoming recurrent weights into each neuron 206 ˆ b = | 1 /N (cid:80) i (cid:80) j J ij | .
This deﬁnition of ˆ b is consistent with the deﬁnition of b in our theory above.
We ﬁnd that the trained networks that are initialized with b = 1 become progressively more tightlybalanced (Figure 3C) over training epochs, as can be seen by a growing ˆ b and by an improvinghigh-frequency encoding (Figure 3E).
We prematurely stop after 100000 training epochs to avoidnumerical instabilities for very tightly balanced networks.
We note that the average of the externalinput weights 1 N (cid:80) i w in i becomes more positive over training epochs (Figure 3C inset, green line)while
the average row sum of the recurrent weights (Figure 3C inset, black line) become morenegative.
Consistent with our theory, we ﬁnd that the mutual information rate (Gaussian channel approximations) grows over training epochs as the network becomes more tightly balanced (Fig 3F).
Notethat we did not train the network to maximize the mutual information rate, but to minimize themean-squared error between target output and actual output, but the two are closely related [28].
Wenote that this tracking does not rely on the threshold-linear input-output function, in fact, we alsosuccessfully trained sigmoid (Fig. 9), linear (Fig. 10) and threshold-quadratic input-output functionφ (Fig. 11) (see appendix H for additional results).
Training on multiple inputs results in tightly balanced subnetworks
We note that networks trained to simultaneously track multiple input Ornstein-Uhlenbeck signals bya linear readout exhibit a spontaneous symmetry breaking into weakly connected subnetworks withstrong local inhibition (Fig. 4 and see appendix F for results on training on multiple signals).
Wenote that this ﬁnding is not limited to training networks on two independent OU-processes; stronglyinhibitory coupled subnetworks also emerge when training on more OU-processes (See Figure 4F forthe resulting connectivity of a network trained on 3 OU-processes).
For networks trained on multipletime-varying stimuli, we observed an interesting training dynamics.
First, they develop a singleglobal inhibitory mode (ﬁgure 7), similar to networks trained to track a single OU-signal.
After sometime, we observe the emergence of a second outlier eigenvalue.
Concomitantly, the network breaksinto two inhibitory subnetworks and the readouts start to track both OU-input signals independently.
Training on nontrivial computation results in tightly balanced network
Finally, we demonstrate the generality of our ﬁndings on two nontrivial tasks where the RNN notonly copies a time-varying input but performs a computation on the input streams.
Firstly, we trained recurrent networks that receive two time-varying input signals to output a continuous extension of the XOR function f
( I 1 ( t ) , I 2 ( t ))
=
| I 1 ( t ) − I 2 ( t ) | , which satisﬁes for I 1 , I 2 ∈ 0 , 1the binary XOR function (Fig 4D).
Consistent with our theory, we observe that the networks developed an outlier eigenvalue with strongly negative real part (Fig 4A).
K-means clustering of therecurrent weight matrix revealed that the RNNs developed tightly balanced subnetworks throughouttraining (Fig 4E).
See appendix F for further details.
Secondly, we trained RNNs to compute a linear transformation of time-varying vector of stochasticprocesses y i ( t ) =
(cid:80)
j
A ij I j ( t ) at every moment in time.
We observed that the RNNs develop anumber of negative outlier eigenvalues and concomitantly the same number of tightly balanced
[Figure]
a
[Figure]
u t s / i n p
[Figure]
s
[Figure]
inhibitory subnetworks that corresponds to the rank of the linear input-output transformation
A (See
Fig 12 and appendix F for further details).
Limitations
For mathematical tractability, we considered ﬁring rate networks here, but for more detailed neuronmodels, other biophysical features shape the information encoding rate.
For example, in spiking neuron models, ﬂuctuating background input can enhance the high-frequency encoding [29].
Moreover,the spike generation mechanism [30, 31, 32, 33, 34, 35], the synaptic dynamics ([36, 37]) and otherlatent variables (e.g. adaptation [38]) shape the frequency-response.
It remains an important futurework to extend the work here by such contributions.
Moreover, moving beyond point-neurons, theshape of the dendritic tree can also affect the frequency response[35].While we show that the dynamics of RNNs trained on several linear and nonlinear tasks is consistent with our DMFT, it remains animportant challenge to consider tasks with more complex temporal structure, e.g. involving memory,where the task dynamics might interfere with the tightness of balance.
Finally, our theory becomesexact only in the limit of large network size, and it is important to also consider ﬁnite-size effects.
For ﬁnite network size, both noise and chaotic ﬂuctuations also contribute to ﬂuctuations of the meanm , which are also recurrently canceled.
Discussion
We show how a tight balance of excitatory currents by recurrent inhibition improves informationencoding of a time-varying signal.
We demonstrate that the mutual information rate of a time-varyingsignal increases linearly with the tightness of balance, both in the presence of additive noise andwith recurrent chaotic network ﬂuctuations.
A non-stationary dynamic mean-ﬁeld theory reveals aseparation of timescale between the mean currents and the timescale of residual ﬂuctuations.
Themean dynamics become linearly faster with the tightness of balanced and enable reliable encoding oftime-varying signals.
In contrast, the timescale of the chaotic dynamics in the residual ﬂuctuationsare largely unaffected by the tightness of balance.
We ﬁnd that networks become more robust todeteriorating effects of ﬂuctuations from noise and chaos as the network becomes more tightlybalanced.
Our study is relevant in the recent debate on the functional implications of how tightly excitatorycurrents are tracked by recurrent inhibition [26, 27, 7, 10, 13, 14].
We address this question by buildinga bridge from information-theoretic measures of information encoding that were previously usedin neuroscience mostly in sensory systems [39, 40, 41, 23, 24] to dynamic mean-ﬁeld descriptionsof recurrent network dynamics that were previously used to describe the often chaotic dynamics ofrecurrent rate networks [18, 19, 20, 16, 42, 17, 43, 14].
Besides the implications on information encoding of more loosely or more tightly balanced networks,of course also biophysical, energetic, and evolutionary constraints should be considered.
Biologicalnetworks can naturally not be arbitrarily tight balanced, which would require arbitrary large synapticcurrents.
Very tight balance might also be questionable for energetic reasons, as was asked previously,"Why should the cortex simultaneously push on the accelerator and on the brake?"
However,such biophysical and energetic constraints may be better addressed in biophysically more detailedmodels.
We found that training networks on tracking a time-dependent signal by a linear readout by minimizingthe squared error make them more tightly balanced, as reﬂected in more negative mean recurrentweights and more positive feedforward input weights.
Moreover, the training also increased themutual information rate between the input signal and linear readout.
This is consistent with ourtheoretical result.
Furthermore, the fact that training arrives at a tightly balanced solution may suggestthat this is a typical solution for a network.
It is an interesting question when such balanced solutionswill emerge with biologically plausible learning rules [45, 46, 47, 48])
The dynamic cancellation of time-varying input through recurrent inhibitory feedback has beenpreviously studied in balanced networks with binary [2, 26, 27] and spiking neurons [49, 50].
Chaos in balanced ﬁring-rate networks was studied previously [19, 20, 51, 17], but the dynamiccancellation of correlated input and its implications on information encoding in rate networks werenot investigated, nor were the implications for training such networks in a machine learning setupwith backpropagation through time.
It would be interesting to extend our mean-ﬁeld analysis to ratenetworks with pre-existing low-rank structures on top of the random structure [52, 53, 54, 55].
The underlying mechanisms of tracking of time-varying input we analyze here are not speciﬁc to fullyconnected threshold-linear RNNs driven by Ornstein-Uhlenbeck signals with additive Gaussian whitenoise, which we merely chose for the sake of simplicity and analytical tractability.
Rate networks withother non-negative input-output transfer functions exhibit a qualitatively similar dynamic tracking inthe tightly balanced regime.
Moreover, the mechanism of tracking described here is tightly related tobalanced predictive coding networks [13, 14].
A testable hypothesis based on our ﬁndings is that changing the tightness of balance, e.g. bypharmacological manipulation or by a genetic knockout that affects net recurrent inhibitory strength(without generating runaway activity or pathological network states), would also affect the highfrequency stimulus encoding and information transmission.
Speciﬁcally, we predict that strongereffective recurrent weights in conjunction with stronger excitatory inputs would improve both thehigh-frequency encoding and the mutual information rate between a stimulus and the populationresponse.
Conversely, when weakening recurrent inhibition and external input strength, we predict animpaired high-frequency encoding and a lower mutual information rate.
Such predictions could notonly be tested in vivo, but also by in vitro experiments, where the number of recurrent synapses canbe manipulated [56].
Checklist
a)
any potential negative
work?
[Yes
, not
review
and
them?
[Table]
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] , see result section and appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] See supplement and here
training-balanced-nets
b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?[Yes] See training section and
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] Error bars for ﬁgure 1E are smaller than symbol size, ﬁgures on mutualinformation density and gain included a comparison to theory as check of error d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See appendix C4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A] Does not apply
[Table]
b) Did you mention the license of the assets?
[N/A]
[Table]
not apply
[Table]
c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Does not
[Table]
apply
[Table]
) Did you discuss whether and how consent was obtained from people whose data you’re us-
[Table]
ing/
[Table]
[N/
[Table]
] Does not apply
[Table]
e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Table]
[Table] [Table]
not apply
[Table]
or subjects
[Table]
[Table] [Table]
[N
[Table] [Table]
not apply
[Table]
you describe any potential to
approvals if
[Table]
[Table] [Table]
not apply
[Table]
you include
[Table] [Table]
participant
[Table]
[N
[Table]
] Does not apply
A Non-stationary dynamic mean-ﬁeld theory with both independent and
common input
The core idea of dynamic mean-ﬁeld theory is that for large N , the distribution of the residual recurrent input for 495 different neurons becomes Gaussian and pairwise uncorrelated, according to the central limit theorem.
To thisend, we characterize the distribution of the ˜ h
i ( t ) (Eq 3b) by considering the (linear) stochastic dynamics:
[Equation]
where η ( t ) is a Gaussian process with mean (cid:104)
η ( t ) (cid:105) = 0 and autocorrelation
[Equation]
Here and in the following, angular brackets denote expectation values over the distribution of the stochasticprocess ˜ h ( t ) , which approximates population averages in the full network.
The mean-ﬁeld estimate for the meanm ( t ) therefore evolves according to Eq 3a with ν ( t ) =
(cid:104) φ ( m ( t ) + ˜ h
( t ))
(cid:105) , the mean-ﬁeld estimate of the meanpopulation ﬁring rate.
We obtain an expression for the time evolution of the two-time autocorrelation function c ( t, s ) =
(cid:68) ˜
h ( t )˜
h ( s ) (cid:69) ,which explicitly depends on two time points.
Taking the temporal derivative of c ( t, s ) with respect to s and 504 using Eq 6, we obtain
[Equation] [Equation]
where r ( t, s ) =
(cid:68) ˜
h ( t ) ( η ( s ) + ξ ( s )) (cid:69) , which we take as an auxiliary function.
Taking the temporal derivativeof r ( t, s ) with respect to t and using Eq 6 again, we arrive at
[Equation]
where q ( t, s )
=
g 2 (cid:10) φ ( m ( t ) + ˜ h ( t )) φ ( m ( s ) + ˜ h
( s ))
(cid:11) (see Eq 7) and σ 2 is the variance of ﬂuctuations drivenby
the independent noise input.
The additional term proportional to δ ( t − s ) in Eq.
9 is due to this independent 509 additive Gaussian white noise, similarly to the independent input considered previously [17, 16].
However, weconsider simultaneously a time-varying common input signal I ( t ) , which renders the problem non-stationary
Together, the dynamic mean-ﬁeld equations for m ( t ) , c ( t, s ) and r ( t, s ) form a closed system of self-consistentdynamic equations and can be solved forward in time s and t by integrating them on a two-dimensional grid from 515 some initial condition for m , c and r .
The integration requires q ( t, s ) , which can be calculated by evaluatinga Gaussian double integral that depends on c ( t, s ) , c ( t, t ) , c ( s, s ) , m ( t ) and m ( s ) .
For the threshold-lineartransfer function φ ( x ) = max( x, 0) , one integral can be evaluated analytically, which allows for an efﬁcientnumerical implementation using adaptive Gauss–Kronrod integration.
The non-stationary dynamic mean-ﬁeld 519 theory captures accurately the time-dependent mean population rate ν ( t ) and the two-time autocorrelationfunction from numerical simulations.
In contrast to previous work that did not include noise [21], we consider here a network where neurons 522 simultaneously receive a time-varying input component
I ( t ) ("signal") that is identical across neurons andan input component ξ i ("noise") that is independent across neurons.
This setup is necessary to meaningfully 524 quantify an information rate for a continuous channel, which requires a signal-to-noise ratio.
Furthermore, we gobeyond the previously published non-stationary dynamic mean-ﬁeld theory [21] by introducing a heuristic term 526 that emulates ﬂuctuating input into the m -equation due to ﬁnite network size N similar to [14] (see appendix B).
We note that in contrast to [14], we do solve the self-consistent set of dynamic mean-ﬁeld equations for m , c and 528 r to calculate dynamic gain, mutual information rate and mutual information rate density.
B Heuristic ﬁnite-size ﬂuctuations
The non-stationary dynamic mean-ﬁeld theory is exact in the limit of large N .
However, for the precise evaluation 531 of the spectral coherence, ﬁnite-size network corrections have to be taken into account.
These corrections areexpected to be of order 1 √ N and therefore have only a marginal effect on the dynamic gain.
The ﬁnite-sizeﬂuctuations can be heuristic accounted for by adding an additional ﬂuctuation term to the equation of m ( t ) 534 (mean-ﬁeld version of Eq. 3a), analogously to previous work [14]:
[Equation]
(10) where the mean-ﬁeld estimate of the mean population ﬁring rate ν ( t ) = (cid:104) φ ( m ( t ) + ˜ h
( t ))
(cid:105) is additionallydriven by the ﬂuctuations ζ ( t ) , a noise term that captures the effective ﬁnite-size ﬂuctuations.
In the case of 537 g = 0 , ζ ( t ) reduces to an additive white Gaussian noise processes (AWGN) with autocorrelation function(cid:104) ζ ( t ) ζ ( t + t (cid:48) )
(cid:105) =
τσ 2 δ ( t (cid:48) ) originating from the mean across neurons of the independent input noise ξ i ( t ) .
For g > 0 , the ﬁnite size noise ζ ( t ) should include the effect coming from the residual ﬂuctuations ˜ h
i ( t ) , whichadds a colored noise component to it (see also [14]).
In contrast to [14], we heuristically approximate both 541 amplitude and timescale of the ﬁnite-size ﬂuctuations from the solution of the stationary DMFT.
C Details on numerical and analytical spectral analysis
We numerically solve the stochastic differential equation Eq.
1 using the Euler–Maruyama method but with a 544 distributionally-exact solution of the Ornstein-Uhlenbeck process for I ( t ) , meaning that I ( t ) is not a solution ofstochastic differential equations and instead is directly generated according to its analytical distributions.
To 546 calculate the spectral coherence, we used the function coherence from the Python package scipy.signal version1 .
7 . 3 with a default Hann window window and default parameters except that we set the length of each segment 548 (nperseg) to 2 11 .
We calculated the integral over the mutual information density 1 − C Iν ( f ) using standardtrapezoidal integration.
As pointed out in the main text, we ﬁnd that the mutual information rate approximately 550 grows linear with tightness of balance b .
In ﬁgure 5, we demonstrate that this linear scaling over several ordersof magnitude is also true for different levels of additive Gaussian white noise σ .
We note that for low noise σ 552 and large b , the mutual information rate saturates which is an effect of ﬁnite time-resolution of the integration ofboth dynamic mean-ﬁeld theory and direct numerical simulations that result in an effective upper bound on the 554 frequencies (see ﬁgure 5), which leads to a deviation from the linear scaling of the mutual information rate.
Weconﬁrm this by deliberately introducing an upper cutoff frequency f cutoff up to which the spectral coherence is 556 integrated:
[Equation]
As expected, changing this frequency cutoff directly affects the mutual information rate for large b (see ﬁgure 6).
For high noise σ and small b , we also observe deviation from the linear scaling.
In that case, for ﬁnite simulationtime t sim signal is so weak that spurious correlations between input and output give the illusion of mutual 560 information.
This lower saturation can be ameliorated by a longer simulation time.
All simulations wereperformed on a laptop and took minutes to hours.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
balance
[Figure]
[Figure] [Figure]
[Figure]
M u t u a l i n f o r m a t
i o n ( b i t / ) noise strength 0.
D Details on analytical approximation of mutual information rate
We observed an approximately linear scaling of the mutual information rate with tightness of balance b .
Where 564 does this linear scaling originate from?
The linear scaling of the mutual information rate can be obtained fromthe boost of the effective timescale of the network in combination with the signal-to-noise ratio that comes from 566 the OU-input signal and the additive Gaussian white noise.
The boost of the effective timescale was alreadynoted earlier [2, 14], but we provided a non-stationary mean-ﬁeld theory for nonlinear rate networks.
The power 568 spectral density of the input signal is given by
[Equation]
S noise ( f ) is the power spectrum of band-limited Gaussian white noise:
[Equation]
In a crude approximation that ignores the nonlinearity of the recurrent network dynamics and the interplaybetween m and ˜ h , we can approximate that the b changes the effective low-pass ﬁltering of the network from thelow-pass ﬁltering coming from the leak-term in the dynamics
[Equation]
(14) to an effective ﬁlter of
[Equation]
Thus, the characteristic timescales of the mean m ( t ) and thus of the population ﬁring rate ν ( t ) is τ/ ( b + 1) .
This results from Eq.
3, when assuming a separation of timescales between the time-varying statistics of m and˜ h .
Thus, in a linear approximation, the magnitude square coherence can be approximated to be
[Equation]
[Figure]
a l i n f o r m a t
i o n ( b i t / )
[Figure]
Same as but with different cutoff frequency f cutoff
Model parameters
N = g = 0 , ∆ t = 1 / 2 12 τ , t sim = 2 6 τ , I
= J
= 1 , τ S = 1 .
τ , σ = 0 .
001 , σ = 0 .
1 , I 1 = 0 .
Using the ﬁrst term of the expansion − log( x − 1)
≈ x + x 2
+ x 3 3 + x 4 4 . . .
this results in
[Equation]
Some algebra gives (setting τ = τ S = 1 )
[Equation]
(18) for b (cid:29) σ 2 /I 1 , i.e., tightly balanced networks and large signal-to-noise ratio this results in a linear scaling ofthe mutual information rate with tightness of balance:
[Equation]
E Details on training setup and additional numerical training experiments
An example implementation in Flux, a machine learning library in Julia is available at training-balanced-nets 583 anonymously.
Brieﬂy, we trained standard rate networks ( Eq. 1) to track an OU-process by a linear readout.
We implemented the stochastic dynamics using the Euler–Maruyama method
We used the default ADAM 585 hyperparameters without any additional ﬁne-tuning and found our results to be robust with respect to choices ofparameters.
parameters of the network simulations are reported in the respective captions.
F Details on training networks to track multiple time-varying signals
We found that training recurrent neural networks simultaneously on multiple signals yields balanced subnetworks.
589 We investigated this in the following scenario:
We trained RNNs to approximate two time-varying external inputs I i ( t ) by two linear readout ˆ I ( t )
= 1 /N (cid:80) i w out i φ ( h i ) , by minimizing the mean squared loss l = 591 (cid:82) | ˆ I ( t ) − I i ( t ) | 2 d t .
The training scenario is identical to training on tracking a single OU-process, the onlydifference is that there exist two independent OU-input processes
I i ( t ) and two readout vectors w out i .
All 593 parameters (recurrent weight matrix J ij , input weights and output weights) are initialized with parameters drawnindependently identically from Gaussian distributions as before.
We found that such randomly initialized networks trained on two OU-processes spontaneously break into 596 two balanced subnetworks throughout training.
The subnetworks have strong local inhibition among neuronsbelonging to one subnetwork, but weak connectivity between the subnetworks.
We displayed that ﬁnding in 598 Figure 4.
We ﬁnd the two subnetworks by k-means clustering on the input weights.
Note that in the case of two
OU-processes, the sorting can also be obtained by simply sorting the weight matrix J ij according to the indices 600 of the sorted input weight vectors, i.e. by performing ’sortperm’ on one of the input vectors using the resultingindices as row- and column indices of the displayed weight matrix J ij .
Furthermore, we found that the entries 602 of the input weights, which were independent Gaussian random initially also displayed symmetry breaking.
We found that the entries of input and output weights associated with the same signal are strongly correlated 604 (Figure 4D), but entries associated with different signals are anti-correlated (Figure 4E).
We note that this ﬁndingis not limited to training networks on two independent OU-processes; strongly inhibitory coupled subnetworks 606 also emerge when training on more OU-processes (See Figure 4F for the resulting connectivity of a networktrained on 3 OU-processes).
For networks trained on multiple time-varying stimuli, we observed an interesting 608 training dynamics.
First, they develop a single global inhibitory mode (ﬁgure 7), similar to networks trained totrack a single OU-signal.
After some intermediate time, we observe the emergence of a second negative outlier 610 eigenvalue.
Concomitantly, the network breaks into two inhibitory subnetworks and the readouts start to trackboth OU-input signals independently.
An analytical analysis of this phenomenon is beyond this publication, but it 612 can be explained in terms of the network to learn subsequent singular values of the input-output cross-correlationmatrix [57], where the singular value that captures most of the variance is learned ﬁrst.
Indeed, tracking the 614 average of the two OU-signals at each moment in time is favorable compared to e.g. tracking only one andignoring the other.
This can be seen by calculating the expectation of the mean squared error for the two cases.
In the case where the mean is perfectly tracked, the mean squared error is by a factor of two larger in case onlyone is tracked and the other is ignored.
[Figure]
to approximate two time-varying external inputs
I i (
) by two linear readout
ˆ
I i ( t ) = 1 /N (cid:80)
j w out ij φ ( h j ) , by minimizing the mean squared loss l = (cid:82)
| ˆ I i ( t ) −
I i ( t ) | 2 d t .
B) When trained on two independent stimuli, two outlier eigenvalues with negative real parts emerge.
C)
When observing the mean connectivity strength within the two subnetworks and between and the global mean coupling, and interesting training dynamics emerges: First, all weights become more negative and one single negative outlier eigenvalue evolves, then at some intermediate state, the network breaks up into two subnetworks, and their respective means become more negative but the weights between them on average actually become closer to zero.).
Model parameters at initialization: N = 100 , g = √ 2 , ∆ t = 0 .
2 τ , t sim = 100 τ, b = 1 , I
= J
= 1 , τ S = 0 .
1 , σ = 1 , epochs = 30000 .
G Mixed excitatory-inhibitory circuits
To corroborate our training results
[Figure]
we also initialized the trained network with separate excitatory and inhibitory 620 populations using the classical parametrization of [1], which for large b results in a balanced state.
Consistentwith [1]
[Figure]
we chose mean couplings J EE =
J IE = 1 .
/N
, J II = − 1 . 8 /N , J EI = − 2 .
/N .
We found that 622 training such a network on the auto-encoder task also results in a single inhibition-dominated population (See
Figure 7).
We did not try to constrain the weight update to maintain a separation of excitation and inhibition.
624 This interesting scenario is left for future work.
[Figure]
H Training networks with other nonlinearities
To corroborate our training results and show the generality of our ﬁndings, we trained networks on the autoencod- 627 ing task also using sigmoid (Figure 9), linear (Figure 10) and threshold-quadratic (Figure 11)
activation functionφ ( x ) .
We found that training such networks on the auto-encoder task also results in a single inhibition-dominated 629 population (See Figure 9 - 11) that becomes progressively more tightly balanced over time, as seen from theoutlier eigenvalue that emerges over time.
I Training on linear transformation on multiple input streams results intightly balanced subnetworks
To furhter corroborate our training results and show the generality of our ﬁndings, we also trained networks on 634 calculating a linear transformation on multiple input streams in each moment in time.
[Figure]
τ, b = 1 , I
= J
= 1 , τ S = 0 .
1 , σ = 1 , epochs = 30000 .
For that, analogous to Fig.4B-D, RNNs are are trained to perform a linear transformation y i ( t ) =
(cid:80)
j
A ij I j ( t )on a set of multi-dimensional time-varying external input
I i ( t ) (Fig 12 top middle)).
The training was done by 637 minimizing the mean squared loss l = (cid:80) j (cid:82) | ˆ y ( t ) − (cid:80) ij A ij I j ( t )
| 2 d t , where ˆ y i are linear readouts of therecurrent network, ˆ y i ( t ) = 1 /N (cid:80)
j w out ij φ ( h j ) and A is a random matrix of size S × R with entries drawnindependently from a Gaussian distribution with zero mean and variance 1 /R .
Here wo chose R = S = 3 .
Again, the time-varying input signals I j ( t ) were independent OU processes.
We observe that the test loss drops throughout training (Fig 12 top left)), concomitantly the linearized network 642 dynamics develops three outlier with negative real part throughout the course of the training (Fig 12 top right)).
We observe that the number of outlier eigenvalues is determined by the rank of A (Fig 12 bottom left)) and 644 their training dynamics seems to be shaped by the singular values of A , reminiscent of the training dynamics ofmulti-layer perceptrons [57].
For our case of R = S = 3 in ﬁgure 12 we observed that three outlier eigenvalues 646 develop, at the same time three tightly balanced subnetworks with strong inhibition emerge (Fig 12 bottom right)).
For illustration, we plotted the activity
h
i for 5 example neurons in the bottom middle of Fig 12.
Consistent 648 with our theory, the network becomes more tightly balanced over the course of training.
We leave an analyticaldescription of the training dynamics for future work.
[Figure] [Figure]
Epoch 1e 101010t
e s t
e rr o r
Time ( )   inputtargetsoutput
Time ( )
e x a m p l e h  real( i ) 0.
i m a g ( i ) initialtrained
Epoch 1e
r e a l ( i )     50   0.
[Table]