A time-resolved theory of information encoding in recurrent neural networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Mammalian brains process information by the collective dynamics of a deeplylayered structure of recurrent networks.
Information transmission in neural circuitsdepends on how well time-varying stimuli are encoded by neural populations.
Adynamic balance of externally incoming currents by strong recurrent inhibitionwas previously proposed as a mechanism to accurately and robustly encode theinformation in a time-varying stimulus, but a full theory was missing.
Here, wedevelop a non-stationary dynamic mean-ﬁeld theory that transparently explains howtight balance of excitatory currents by recurrent inhibition improves informationencoding.
We demonstrate that the mutual information rate of a time-varyinginput increases linearly with the tightness of balance, both in the presence ofadditive noise and with recurrent chaotic network ﬂuctuations.
We corroboratedour ﬁndings in numerical experiments and demonstrated that recurrent networkswith positive ﬁring rates trained to transmit a time-varying stimulus generically userecurrent inhibition to increase the information rate.
We also found that networkstrained to transmit multiple independent time-varying signals spontaneously formmultiple local inhibitory clusters, one for each input channel.
Our ﬁndings suggestthat feedforward excitatory synaptic projections and local recurrent inhibition - asobserved in many biological circuits - is a generic circuit motif for encoding andtransmitting time-varying information in recurrent neural circuits.
Introduction
How fast and reliable time-varying incoming stimuli are encoded in the population activity ofrecurrent neural networks constrains information transmission between local circuits and inter-arealcommunication.
While the brain has to respond rapidly and reliably to changes in the world, the timescales of synapses can be slow, single neurons exhibit Poisson-like temporally irregular dynamicsand dynamic instability of recurrent neural circuit dynamics can be a source of chaotic variability[1, 2, 3, 4, 5].
Excitation and inhibition in biological neural networks is usually conveyed by different types ofneurons with a predominance of recurrent inhibitory feedback, a property known as ’inhibitiondominance’ [6, 7, 8, 9].
Networks with very strong recurrent inhibition (where synaptic interactionsscale O (cid:16) 1 / √ N (cid:17) with network size N ), a dynamic balance arises between excitatory currents andrecurrent inhibitory currents that was originally proposed as a robust mechanism to describe theemergence of asynchronous irregular activity[1].
Binary networks in such a balanced state track time-varying signals [2].
While subsequent experimental work found evidence for a dynamic balance of input currents, the ’tightness’ or ’looseness’ ofsuch a dynamic balance in cortical circuits, as well as the computational implications are a question of active scientiﬁc debate [10]. ’
Loosely’ balanced networks, where excitatory and inhibitory currentsare O (1) compared to the distance from reset to threshold, can respond in their population ﬁring ratenonlinearly to external inputs, which was argued to be favorable for sensory processing [7].’Tightbalance’ refers to a more precise tracking of total excitatory and total inhibitory input currents in time.
Such ’tight balance’ was also studied in a series of works that arrived at spiking balanced networksfrom a normative predictive coding ansatz [11, 12, 13, 14].
It is important to investigate how suchbiological features such as ’tightness of balance’ shape the reliability of population response to external stimuli.
How the dynamic tracking of time-dependent inputs described in binary networks [1, 2]extends to ﬁring rate models and how it affects the information encoding of time-varying stimuli inthe presence of chaos and noise has not yet been addressed.
Previous dynamic mean-ﬁeld theory(DMFT) approaches to input-driven rate networks assumed that the mean of the external inputs acrossneurons does not depend on time, which facilitates DMFT [15, 16, 17, 14], but does not permit toinvestigate encoding of dynamic stimuli in the population ﬁring rate.
To address this gap, we study how a time-varying stimulus is encoded in the population rate of aninhibition-dominated rate network under the inﬂuence of additive chaos and noise.
We show that the accuracy of neuronal population encoding improves in more tightly balancedrecurrent networks because of a speedup of the effective time scale of the population mean dynamics.
Conventional methods of dynamic mean-ﬁeld theory [18, 19, 20, 17] are not adequate to capture theeffects of time-varying common input.
Therefore, we developed a dynamic mean-ﬁeld theory that isnon-stationary, meaning that the order parameters are time-dependent (Materials and Methods).
Beyond a similar recent approach [21], our theory can treat both time-varying common and independentexternal input, which is crucial to analytically treat a signal-to-noise ratio that constrains populationcoding.
This novel technique accurately captures the time-dependent mean, variance and two-pointautocorrelation of the input-driven networks.
Speciﬁcally, we calculate the cross-spectral densitybetween input and output and the power spectral density of the population ﬁring rate.
Together withthe knowledge of the input statistics, this allows us to calculate the mutual information betweenstimulus and population rate in the Gaussian channel approximation based on the spectral coherence.
We examine how the frequency-response and mutual information rate depends on tightness of balance,added noise, network chaos and statistics of the input stimulus, using both theory and simulation.
Allthe analytic results match those from network simulations.
We show that recurrent networks that aretrained on tracking a time-dependent stimulus develop strong recurrent inhibition and strong, positiveinput weights, a ﬁngerprint of the balanced state.
Concomitantly, the mutual information rate betweentheir network readout and the stimulus increases as predicted by our theory.
This indicates that a moretightly balanced state is a generic solution to reliable transmitting a time-varying input in the presenceof noise or chaos.
Lastly, we ﬁnd that networks trained on simultaneously transmitting multipleindependent time-varying stimuli spontaneously break up into weakly connected subnetworks withstrong local inhibition.
Our ﬁndings have important implications for information encoding in ﬁring rate networks and forunderstanding how neural network architecture design shapes noise-robustness and informationencoding.
Population coding in recurrent networks
We study how well a time-varying scalar input x ( t ) is encoded in the population ﬁring rate φ ( h ( t ))of a recurrent network of N nonlinear rate units (’neurons’) that obey
[Equation]
(1) with each entry of the coupling matrix J ij = − J 0 /N
+ ˜ J
ij drawn from a Gaussian distribution withnegative mean − J 0 /N and variance g 2 /N , where g is a gain parameter that controls the heterogeneityof weights.
The transfer function φ is set to a threshold-linear function φ ( x ) = max( x, 0) .
Thetime-varying input signal x ( t ) is decoded from the population ﬁring rate φ ( t ) by a linear readoutˆ x ( t ) = 1 /N (cid:80) i w out i φ ( h i ) . The external input contains a signal component I ( t ) , which is identicalacross neurons and a noise term ξ i ( t ) , which is independent across neurons. For concreteness, here wechoose ξ i to be independent additive white Gaussian noise processes (AWGN) with autocorrelation
I n p u t I ( t ) A A   1. P o p u l a t i o n r a t e
B
{+B 10 0 10 1 
+}
{+Frequency (1/ ) 
+}
C o h e r e n c e
{+D 101010balance b 
+}
M u t u a l i n f o r m a t
i o n ( b i t / )
E theory simulations
Time ( ) 0.
P o p u l a t i o n r a t e
C
C function (cid:104) ξ i ( t ) ξ i ( t + t (cid:48) )
(cid:105) =
τσ 2 N δ ( t (cid:48) ) and the signal I ( t ) to be an Ornstein-Uhlenbeck process
[Equation]
The amplitude of the AWGN ξ S ( t ) is denoted by I 1 .
b is a parameter that multiplies both external inputand recurrent coupling strength and thus regulates the tightness of balance.
For ﬁring-rate networks,the ability to track time-dependent input strongly depends on the strength of balance b (Fig 1), whichdetermines how closely time-varying input is tracked by the population rate.
To understand howthis tracking arises in the model, it is useful to rewrite Eq 1 by decomposing h i
= m
+ ˜ h
i intoneuron-speciﬁc and neuron-nonspeciﬁc components and writing J ij = − J 0 /b
+ ˜ J ij .
This results in
[Equation]
Here δI ( t ) directly enters the expression for m , because it is identical across all neurons.
It thusdirectly impacts the mean population rate ν ( t ) = 1 N (cid:80) i φ ( h i ( t )) and recruits, through the negativerecurrent mean coupling − J 0 /b , strong recurrent feedback − bJ
ν ( t ) that is anticorrelated with theinput and cancels most of the common external input I ( t ) .
This cancellation can be seen by rewriting Eq 5a as
[Equation]
This equation is commonly referred to as the ’balance equation’ [2, 19, 20] in the absence oftime-dependent input.
On the other hand, the noise ξ i affects only the residual ﬂuctuations ˜ h .
As the balance parameter b is increased, the effective timescale of the dynamics of m becomes shorterby a factor of b , which leads to faster and more precise tracking. This can be seen by rewriting [Equation]
In contrast, the timescale of the equation for ˜ h
i is unchanged by b .
Of course, the two equationsfor m and d˜ h i are interdependent via φ ( h i ) =
φ ( m + ˜ h i ) , so for a complete treatment, the jointdynamics of m and ˜ h i has to be solved.
The basic idea of DMFT is that for large N , the distribution of the recurrent input for differentneurons becomes Gaussian and pairwise uncorrelated, according to the central limit theorem. To thisend, we characterize the distribution of the ˜ h i ( t ) by considering the (linear) stochastic dynamics:
[Equation]
6) where η ( t ) is a Gaussian process with mean (cid:104) η ( t ) (cid:105) = 0 and autocorrelation [Equation] Here and in the following, angular brackets denote expectation values over the distribution of thestochastic process ˜ h ( t ) , which approximates population averages in the full network.
The mean-ﬁeldestimate for the mean m ( t ) therefore evolves according to Eq 5a with ν ( t ) =
(cid:104) φ ( m ( t ) + ˜ h
( t ))
(cid:105) , themean-ﬁeld estimate of the mean population ﬁring rate.
We obtain an expression for the time evolution of the two-time autocorrelation function c ( t, s ) = 114 (cid:68) ˜
h
( t )˜
h
( s ) (cid:69) , which explicitly depends on two time points.
Together, the dynamic mean-ﬁeld equationsfor m ( t ) , c ( t, s ) and an auxiliary function r ( t, s ) = (cid:68) ˜
h ( t ) η ( s ) (cid:69) form a closed system of selfconsistent dynamic equations and can be solved (For details of the nonstationary dynamic mean-ﬁeldtheory, see appendix A).
In contrast to previous work [21], we consider here a network where neuronssimultaneously receive a time-varying input component
I ( t ) ("signal") that is identical across neuronsand a input component ξ i s ("noise") that is independent across neurons.
This setup is necessaryto meaningfully quantify an information rate for a continuous channel, that requires a signal-tonoise ratio.
We note that we go beyond a previously published nonstationary dynamic mean-ﬁeldtheory [21], by introducing in the time-dependent mean-ﬁeld theory a heuristic term that emulatesﬂuctuating input into the m -equation due to ﬁnite network size N similar to [14] (For details of theheuristic ﬁnite size approximation, see appendix B).
We note, that in contrast to [14], we do solve theself-consistent set of dynamic mean-ﬁeld equations for m , c and r to calculate dynamic gain, mutualinformation rate and and mutual information rate density.
Frequency-dependent signal encoding in population response
We next systematically analyze how different input frequency are encoded in the population responseand observe that the transmission of high frequencies stimuli is improved for large b (’tightly balanced’regime) (Figure 2A and C).
We calculate dynamic gain G ( f ) =
| S xy | / | S xy | of the recurrent networksboth in direct numerical simulations and using the nonstationary DMFT.
This quantiﬁes how in linearresponse a variation in the input signal I ( t ) affects the population rate ν ( t ) and is a standard analysis
[Figure]
a [Figure] n
[Figure] [Figure] t [Figure]
[-I 
-]y
[Figure] i t / [Figure] [Figure]
DDD
{+= 0 for A) and B) . g = 2 for C) and D) , ∆ t = 1 / 2 10 /τ , t sim = 2 6 τ , I 
 = J 
 = 1 ,+} τ S = 1 .
τ , σ = 0 .
1 , I 1 = 0 .
for characterizing the response properties of individual cells [22].
S xy is the Fourier transformationof the input-output crosscorrelation function between I ( t ) and ν ( t ) .
S xx is the power spectral densityof the OU-input.
We calculate G ( f ) both using direct numerical network simulations and using dynamic mean ﬁeldtheory (For details of the spectral analysis, see appendix C). The Fourier transform of the autocorrelation of the input is the power spectral density of the input, according to the Wiener-Khinchin theorem.
For an Ornstein-Uhlenbeck process, the power spectral density is S xx ( f ) = [Equation]
We observe that increasing b boosts the dynamic gain (Figure 2A and C for g = 0 and g = 2 ).
Thisis a direct consequence of the speedup of the dynamics of m -dynamics as seen in Eq.
Thus, asthe network becomes more tightly balanced, high-frequency signals can be encoded and transmittedmore reliable.
To quantify that improved information encoding in terms of bit per τ we turn to ainformation theoretic analysis in the next section.
Mutual information rate between time-varying stimulus and population
response
With information theory we can treat the recurrent neural network as a noisy communication channeltransforming a signal embedded in a noisy current into a population rate that can be readout byanother population. The mutual information rate measures how much the uncertainty about the input I ( t ) is reduced given the population rate ν ( t ) per unit time τ : [Equation]
where p ( x, y ) is the joint probability density function of x ( t ) and y ( t ) , h ( X ) and h ( Y ) are theirentropy rates and h ( X | Y ) is the conditional entropy rate of x ( t ) given y ( t ) .
The continuous Gaussianchannel gives a lower bound on the mutual information rate for a Gaussian input signal: [23, 24]: [Equation]
The inequality results from the property that a Gaussian process has the maximum entropy of allprocesses with ﬁxed variance.
Recently it was shown for spiking neurons that for moderate input modulation in a ﬂuctuation driven regime this lower bound is very close to the mutual information rateestimated from direct methods [25]. . This is convenient, because estimating the mutual informationfrom empirical data is computationally costly, as the sample size has to be much larger than the sizeof the alphabet [26]. . Continuous processes usually have to be discretized resulting in very largealphabets. In our case a Gaussian channel approximation to the mutual information rate betweeninput current and output population rate was estimated based on the spectral coherence between inputcurrent and output spike train, which is based purely on second-order statistics, which we can obtainthrough the non-stationary dynamic mean-ﬁeld theory
[Equation] C xy ( f ) denotes the magnitude squared spectral coherence.
The spectral coherence is the frequencydomain analog of correlation and measures the linear relationship between frequency components ofinput and output signal.
Its magnitude square is [Equation]
It consists of S xx ( f ) ,which is the power spectrum of the input signal (a OU process), S yy ( f ) isthe power spectrum of the population rate ν ( t ) .
S yy
( f ) = lim T →∞ 1 T (cid:104) ˜ y ˜ y ∗ (cid:105) , where ˜ y ( f ) is the
Fourier transform of the population ﬁring rate ν . If and only if x ( t ) and y ( t ) are linearly scaledcopies of each other, then C xy ( f ) =1. If x ( t ) and y ( t ) are independent, then C xy ( f ) =0, the reversegenerally does not hold.
Nonlinear effects and noise reduce the coherence.
We calculated the spectralcoherence in two independent ways: Firstly based on the nonstationary DMFT and secondly basedon direct numerical simulations.
The dependence of C xy ( f ) on the depth of balance b are depicted in Figure 1D. DMFT and direct numerical simulations with additive band-limited white noise ξ ( t ) arein excellent agreement.
The mutual information rate scales approximately linear with b (See ﬁgure 1E).
For large b , the mutualinformation rate saturates. The saturation level is determined by the band limit of the incoming signal(See ﬁgure in appendix C).
We use the analytical high-frequency behavior and low-frequency limit togive an analytical approximation of the linear scaling of the mutual information rate (See appendix C). From the low-frequency limit and the high-frequency limit of the coherence, we can get an analyticalestimate of the spectral coherence. The high-frequency behavior strongly depends on the depth ofbalance. We analytically approximate the transition points from the low-frequency response to thehigh-frequency response and stitched analytic high- and low-frequency estimates together at thetransition points.
Plugging into Eq.
10 results in a linear scaling of the mutual information rate with b(see appendix D for derivation).
We conclude that the mutual information rate scales approximately linearly with tightness of balanceb for sufﬁciently high band limit.
Where does the linear scaling of the mutual information rate withtightness of balance arise from?
The reason is that the b determines the cutoff frequency, up to whichthe network transmits information faithfully.
Contributions from frequencies beyond this cutoff arenegligible to ﬁrst order.
This conclusion is not restricted to networks of rectiﬁed linear units, a similarline of argumentation holds also e.g. for the threshold-powerlaw and other models with non-negativeﬁring rates (See e.g. [19, 20] for balanced rate networks with other input-output transfer functions).
Inspired by a recent work on the frequency-resolved information encoding rate in single cells [25],we analyze for the population rate of recurrent neural networks how how the encoding of differentfrequency components of the input signal depend on the tightness of balance b , the strength of addedweight noise σ and the heterogeneity of the recurrent weights g .
For that, we considered the mutual information rate density r ( f ) , which gives a frequency-resolvedquantiﬁcation of the information encoding rate (See for details of deﬁnition [25]).
r lb ( X, Y ) =− log 2 (1 − C xy ( f )) Note that the integral over frequencies of r lb ( X, Y ) gives the Gaussian channelapproximation of the mutual information rate, we thus call it mutual information rate density andremark that it comes in units of bit/τ 1 /τ .
We observe that more tightly balanced networks can transmit more high-frequency information(Figure 2B and D for g = 0 and g = 2 ).
Overall, the information rate for more tightly balancednetworks is increased because of the boost of the signal to noise ratio coming from the fact thatthe signal I ( t ) is grows by b , while the noise is independent of b .
The above described frequencydependent effect comes on top of that.
For recurrent networks with weight heterogeneity ( g > 0 ), we observe a that the residual slow ﬂuctuations ˜ h become stronger which leads to dip in the mutual information density for low frequencies(Compare Figure 2B for g = 0 and D for g = 2 ).
This comes from the fact that g > 0 leads do additional recurrent low-frequency ﬂuctuations thatact as an additional source of noise.
But because these ﬂuctuations are low-frequency, they have adifferent effect than additive Gaussian white noise (See also [14]).
For more tightly balanced networks (larger b ), the residual ﬂuctuations do not impair informationencoding (see darker green lines in ﬁgure 2B), because they are canceled by fast tracking.
A relatedphenomenon was observed in densely connected recurrent binary networks ([27, 28]).
That varianceoriginating from weight heterogeneity ( g > 0 ) is more effectively reduced by balanced networksas function of b compared to variance originating from added Gaussian white noise σ was alreadyobserved in rate networks with static input [14].
Training networks on auto-encoding results in strong inhibition
[Figure]
We corroborated the ﬁnding that the tightness of balance has a crucial role in information encodingin trained recurrent networks which demonstrates the generality of our ﬁndings.
For that, we trainrecurrent networks on an auto-encoder task.
We initialize small-sized recurrent networks ( N = 100 )that follow the dynamics of Eq.
1 in a loosely balanced state ( b = 1 ), e.i., with Gaussian connectivity
J ij that has a negative mean of b/N and variance g 2 /N with g = sqrt 2 , input weights w in andoutput weights w out are drawn from a Gaussian distribution with a positive mean.
The networkreceives the time-varying input signal I i ( t ) =
w in i
I ( t ) , so in contrast to our theoretical setup, inprinciple each neuron can receive the input signal I ( t ) with a different scale w in i .
We then trainedinput weights w in , output weights w out and recurrent weights J ij using backpropagation-through-timewith the ADAM optimizer with standard hyper parameters (see appendix G for additional details ontraining setup and additional controls of training results).
We minimize the mean squared loss meansquared loss l = (cid:82) t | y ( t ) − I ( t ) | 2 d t between time-varying input signal I ( t ) and a linear readoutˆ x ( t ) = 1 /N (cid:80) i w out i φ ( h i ) .
Initially, the network does not track well the time-varying input signal as expected from our theoreticalresults.
Over training epochs, we however ﬁnd that the network learns to track the time-varyinginput (Figure 3A. Analyzing the eigenvalues of the trained network, we observe an outlier eigenvaluewith very negative real part after training that indicates the emerging fast tracking (Figure 3B. Thisis conﬁrmed by measuring the empirical level of balance b in the connectivity, which we quantifyjust by the absolute value of the mean of the sum of incoming recurrent weights into each neuronb = | 1 /N
(cid:80) i (cid:80) j J ij | .
We ﬁnd that the mean becomes progressively more balanced (Figure 3C overtraining epochs.
We prematurely stop after 100000 training epochs to avoid numerical instabilitiesfor very tightly balanced networks.
Consistent with our theoretical results, we ﬁnd that the mutual information rate quantiﬁed in the
Gaussian channel approximations grows over training epochs.
Note that we did not train the networkto maximize the mutual information rate, but to minimize the mean-squared error between targetoutput and actual output.
We note that this tracking does not rely on the threshold-linear input-outputfunction, in fact we also successfully trained threshold-quadratic and sigmoidal input-output functionφ .
We note that training with a linear input-output function φ ( x ) = x fails, because the boost ofhigh-frequency input through the tracking observed in tightly-balanced networks requires that theﬁring rate units have non-negative ﬁring rates (see appendix G for additional controls of trainingresults, e.i. training on sigmoid, threshold quadratic, and the role of additive noise and regularizationof weights and ﬁring rates).
We note that networks trained to simultaneously track multiple input Ornstein-Uhlenbeck signalsby a linear readout exhibit a spontaneous symmetry breaking weakly connected subnetworks withstrong local inhibition (see appendix H for results on training on multiple signals).
Limitations
For mathematical tractability, we considered ﬁring rate networks here, but for more detailed neuronmodels, other biophysically features shape the information encoding rate.
For example, in spiking neuron models, ﬂuctuating background input can enhance the high-frequency encoding [29].
Moreover, the spike generation mechanism [30, 31, 32, 33, 34, 35],the synaptic dynamics and latentvariables (e.g. adaptation [36]) and the synaptic timescale ([37, 38]) shape the frequency-response.
Itremains an important future work to extend the work here by such contributions.
Moreover, movingbeyond point-neurons, the shape of the dendritic tree can also affect the frequency response[35]. While we demonstrated that recurrent networks trained on a simple copy task can be explained byour non-stationary dynamic mean-ﬁeld theory, it remains an important challenge to consider morecomplex tasks, e.g. involving memory, where the task dynamic might interfere with the balance.
Finally, our theory only rigorously describes the behavior of very large networks, and it is importantto also consider ﬁnite-size effects.
For ﬁnite network size, chaotic ﬂuctuations also contribute toa ﬂuctuation of the mean, which are also recurrently canceled.
This was described previously inrecurrent networks in a linear regime.
Discussion
We show how tight balance of excitatory currents by recurrent inhibition improves informationencoding of a time-varying signal.
We demonstrate that the mutual information rate of a time-varyingsignal increases linearly with the tightness of balance, both in the presence of additive noise andwith recurrent chaotic network ﬂuctuations.
A non-stationary dynamic mean-ﬁeld theory revealsa separation of time-scale between the mean currents which become linearly faster with tightnessof balanced and enable reliable encoding of time-varying signals.
In contrast, the time-scale of thechaotic dynamics in the residual ﬂuctuations are largely unaffected by the tightness of balance.
Weﬁnd that networks become more robust to deteriorating effects of ﬂuctuations from noise and chaosas the network becomes more tightly balanced.
Our study is relevant in the recent debate on the functional implications of how tightly excitatorycurrents are tracked by recurrent inhibition [27, 28, 7, 10, 13, 14].
Our work addresses this question by building a bridge from information theoretic measures ofinformation encoding that were previously used in neuroscience mostly in sensory systems [39, 40,41, 24, 23] to dynamic mean-ﬁeld descriptions of recurrent networks dynamics that were previouslyused to describe the often chaotic dynamics of recurrent rate networks [18, 19, 20, 16, 42, 17, 43, 14].
Besides the implications on information encoding of more loosely or more tightly balanced networks,of course also biophysical, energetic and evolutionary constrains should be considered.
Biologicalnetworks can for biophysical reasons not be arbitrarily tight balanced, which would come witharbitrary large synaptic currents.
Too tight balance might also be questionable for energetic reasons,as was asked previously "Why should the cortex simultaneously push on the accelerator and onthe brake?"
However, such energetic and biophysical constrains might be better addressed inbiophysical more detailed models.
We found that training networks on tracking a time-dependent signal by a linear readout by minimizingthe squared error make them more tightly balanced, as reﬂected in more negative mean recurrentweights and more positive feedforward input weights.
Moreover, the training also increased themutual information rate between input signal and linear readout.
This is consistent with our theoreticalresult.
Moreover, the fact that training arrives at a tightly balanced solution seems to suggest that thisis a typical solution for a network.
(See for effect of different regularization of activity and weightsand effect of noise appendix G) Previous studies on temporal information encoding in rate networks were limited to independentinputs across neurons in the form of stochastic [15, 17] and sinusoidal [16] drive, but the networkswere not balanced, and their connectivity had zero mean coupling. In these previous studies, thedistribution of inputs across the population is time-independent [15, 16, 17] and stationary dynamicmean-ﬁeld theory was sufﬁcient to describe the results. However, the treatment of common input isonly possible by the non-stationary dynamic mean-ﬁeld approach introduced here.
The dynamic cancellation of time-varying input through recurrent inhibitory feedback has beenpreviously studied in balanced networks with binary [2, 27, 28], and spiking neurons [45, 46].
Chaos in balanced ﬁring-rate networks was studied previously [19, 20, 47, 17], but the dynamiccancellation of correlated input and its implications on information encoding in rate networks werenot investigated, nor were the implications for training RNNs in a machine learning setup withbackpropagation through time.
It would be interesting to extend our mean-ﬁeld analysis to ratenetworks with pre-existing low-rank structure on top of the random structure [48, 49, 50, 51].
The underlying mechanisms of tracking of time-varying input we analyze here are not speciﬁc tofully-connected threshold-linear RNNs driven by Ornstein-Uhlenbeck signals with additive Gaussianwhite noise, which we merely chose for the sake of simplicity and analytical tractability.
Ratenetworks with other input-output transfer function that are constrained to be non-negative, exhibit aqualitatively similar dynamic tracking in the tightly balanced regime.
Moreover, the mechanisms oftracking described here is tightly related to coding balanced networks [13, 14].
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work? [Yes] See Section (c) Did you discuss any potential negative societal impacts of your work? [Yes] , not applicable
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] , see result section and appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] See supplement(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?[Yes] See training section and supplement
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars for ﬁgure 1E are smaller than symbol size, ﬁgures on mutual information density and gain included comparison to theory as check of error
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See appendix C
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Does not apply
(b) Did you mention the license of the assets?
[Yes] Does not apply
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Does not
apply
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] Does not apply
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
tion or offensive content?
[Yes] Does not apply
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] Does not apply
(b) Did you describe any potential participant
[Table]
A
Nonstationary dynamic mean-ﬁeld theory with both independent and
common input
B Heuristic ﬁnite size correction
C Details on numerical and analytical spectral analysis
D
Details on analytical approximation of mutual information rate
E Finite-size correction for spectral coherence quantiﬁcation
[-N = g = 0 , ∆ t = 1 / 2 12 τ , t sim = 2 6 τ , I 
-]
[-= J 
-]
[-= 1 , τ S = 1 . 
-]
[-≈ x + x 2 
-]
F
[-2 τ , t sim = 100 τ, b = 1 , I 
-]
[-= J 
 = 1 , τ S = 0 . 1 , σ = 1 , epochs = 30000 . G-] Mixed excitatory
[-J IE = 1 . 
 /N , J II = − 1 . 8 /N , J EI = − 2 . 
 /N .-]inhibitory
G
Details ond training setup and
additional numerical training experiments
H
Details on
training networks to track multiple time varying signals