Original: Efficient coding explains neural response homeostasis and stimulus-specific adaptation
Revised: Efficient coding explains neural response homeostasis and stimulus-specific adaptation

Original: Abstract
Revised: Abstract

Original: Empirical studies have demonstrated that across changes in their sensory environment or input statistics, cortical neurons display a homeostasis or equalisation offiring rates [2, 10, 8, 7].
Revised: Across changes in their sensory environment or input statistics, cortical neurons display a homeostasis of firing rates [2, 10, 8, 7].

Original: We present a normative explanation of such firing ratehomeostasis grounded inefficient coding theory and the infomax principle [6].
Revised: We present a normative explanation of such firing-rate homeostasis grounded in the infomax principle [6].

Original: Wefurther demonstrate how homeostatic coding, coupled with Bayesian theories ofneural representation [11] can explain stimulus-specific adaptation effects, whichis widely observed in the nervous system ( e g. , in the visual cortex [2]) and how itcan be achieved by divisive normalisation with adaptive weights [3, 12]
Revised: We further demonstrate how homeostatic coding, coupled with Bayesian theories of neural representation [11] can explain stimulus-specific adaptation effects widely observed in the nervous system ( e.g. , in V1 [2]). This can be achieved by divisive normalisation with adaptive weights [3, 12]. Consider a neuron acting as a feature detector.

Original: Problem statement and framework
Revised: If the feature becomes more prevalent, the neuron responds more often and has a higher average firing rate. Prolonged exposure to such an environment causes the neuron to adapt by responding less vigorously [2, 10, 8].

Original: s h ( s ) Increasing g g Metabolic
Revised: If such adaptation returns the average firing rate to its value prior to environmental shift, it is termed firing rate homeostasis .

Original: Energy E
Revised: This paper uses efficient coding theory to give a general normative account of firing rate homeostasis. We show that homeostasis optimises a trade-off between the information conveyed by the neural response and the metabolic cost of the response. We apply this result to Bayesian theories of representation, showing that this can lead to stimulus specific adaptation effects observed in V1 [2].

Original: Information I
Revised: We consider a population of N neurons, responding to the (possibly high-dimensional) stimulus s , with marginal distribution Z ( s ) that can vary across environments. Assuming rate coding in time bins of a fixed duration, we denote the population spike counts in a coding interval by n = ( n 1 , . . . , n N ) . Neural tuning curves, h i ( s ) , are factorised into a representational curve , Ω i ( s ) , and a gain , g i : h i ( s )

Original: L =
Revised: =

Original: I − E g opt
Revised: g i Ω i ( s )

Original: Submitted to Information- Principles in Systems Workshop at the 36th Conference on Neural
Revised: = gain × representational curve Approximating the spike counts as continuous, we adopt a Gaussian noise model with unit Fano factor: conditional on s , n i are independent with n i | s ∼ N

Original: Systems (NeurIPS 2022)
Revised: ( h i ( s ) , h i ( s )) .

Original: We suggest that neurons adapt their gains(Fig. 1) to maximise an objective function, L 0 ( g ) , thattrades off the metabolic cost of neural activity with the information conveyed by the population [5, 4]:
Revised: We suggest that neurons adapt their gains to maximise an objective function, L 0 ( g ) , that trades off the metabolic cost of neural activity with the information conveyed by the responses [5, 4]:

Original: [Equation]
Revised: [Equation]

Original: Here, I ( s , n ) is the mutual information between the stimulus and response, the second term penalisesthe average population spike count, and µ is a positive constant controlling the information-energytrade-off.
Revised: Here, I ( s , n ) is the mutual information between the stimulus and response, the second term penalises the average population spike count, and µ > 0 controls the information-energy trade-off.

Original: Note that Ω i (which in our framework constitute input data for the gain optimisation problem) canbe complex functions of the stimulus, representing any general computation.
Revised: Note that Ω i can be complex functions of the stimulus ( e.g. multimodal, discontinuous), representing any general computation.

Original: Allowing for thefree adjustment of gainsdoes not restrict the computations  performed by this population either,as adjustments in gains canin principle always be compensated for by reciprocal adjustments inreadout weights.
Revised: Adjusting gains the computations which can be performed by downstream populations, as adjustments in gains can be compensated for by reciprocal adjustments in readout weights.

Original: Thus, our framework can apply toneural populations located deep in the sensoryprocessing pathway which code for high-level latent variables.
Revised: Thus, our framework applies to populations located deep in the sensory processing pathway.

Original: In this sense our treatment is moregeneral than other efficient coding frameworks ( e.g. [4]), which place tight constraints on theneuralrepresentation, via constraints on the shape and configuration of the tuning curves.
Revised: This makes our treatment more general than other efficient coding frameworks ( e.g. [4]), which place tight constraints on the shape and configuration of the tuning curves.

Original:  Information-energy trade-off leads to homeostasis
Revised:  Information-energy trade-off leads to homeostasis

Original: Analytic maximisation of mutual information is intractable.
Revised: Analytic maximisation of mutual information is intractable.

Original: Wetherefore make two approximationsto allow us to derive a closed form expression for the optimal gains (see App. A.1 for details).
Revised: We make two approximations to derive a closed form expression for the optimal gains (see App. A.1 for details).

Original: Forthe Gaussian noise model, the conditional entropy is given by 
Revised: For the Gaussian noise model, the conditional entropy is given by 2 H [ n | s ] = (cid:80) Nj =1 ln( g i ) + const.

Original: [Equation]
Revised: in g .

Original: The marginal entropy, H [ n ] , however, is intractable, and we replace it by an upper bound, namelythe entropy of a Gaussian with the same covariance:
Revised: The marginal entropy, H [ n ] , however, is intractable, and we replace it by an upper bound, namely the entropy of a Gaussian with the same covariance:

Original: [Equation]
Revised: [Equation]

Original: Using I ( s , n ) =

Original: H [ n ] − H [ n | s ] , this yields the upper bound 2 I ( n , s ) ≤ln (det ( I + P diag( g ⊙ ω )))

Original:  + const.
Revised: )) + const.

Original: in g , where⊙ denotes element-wise product, and wedefine
Revised: in g , where

Original: [Equation]
Revised: [Equation]

Original: Here Corr[ · ] denotes the Pearson correlation matrix, and CV i is the coefficient of variation of Ω i ( s)(or equivalently h i ( s ) ).
Revised: and ⊙ denotes element-wise product, Corr[ · ] denotes the Pearson correlation matrix, and CV i is the coefficient of variation of Ω i ( s ) .

Original: This in turn yields (up to  additive constants) the upper bound
Revised: Up to an additive constant, we obtain the upper bound

Original: [Equation]
Revised: [Equation]

Original: which we will optimise  (see App.
Revised: We subsequently optimise L (see App.

Original: A.2 for details).
Revised: A.2 for details).

Original: The maximum satisfies

Original: d Ldg i =
Revised: [Equation] L

Original: µ g i ( I +( P diag( g ⊙ω )) − 1 ) − 1 ii − ω i= 0 .
Revised: [Equation] − 1 ) − 1 ii − ω i .

Original: Under the condition (see below for empirical justification)
Revised: Under the condition [Equation]

Original: [Equation]
Revised: [Equation]

Original: a first-order Neumann expansion in ( P diag( g ⊙ ω )) − 1 yields
Revised: a first-order Neumann expansion in ( P diag( g ⊙ ω )) − 1 yields

Original: d L dg i
Revised: d L dg i

Original: ≈ µg i (1 − [ P − 1 ] ii ω
Revised: ≈ µg i (1 − [ P − 1 ] ii ω

Original: i g i ) − ω
Revised: i g i ) − ω

Original: i .
Revised: i .

Original: Settingthis to zero and reusing Eq.
Revised: Setting this to zero and reusing Eq.

Original: (6) yields the following first-order approximation for the optimal gains
Revised: (5) yields the following first-order approximation for the optimal gains

Original: [Equation]
Revised: [Equation]

Original: The corresponding zeroth-order solution, g (0) i ≡ µ/ω i , gives exact homeostasis , as under these gainsaverage rate of neuroni =
Revised: The corresponding zeroth-order solution, g (0) i ≡ µ/ω i , gives exact homeostasis , as under these gains average rate of neuron

Original: E [ h i ( s )]
Revised: [Equation]

Original: = g (0) i ω i

Original: = µ which is constant, both between neurons and across environments (as specified by Z ( s ) ).
Revised: which is constant, both between neurons and across environments (as specified by Z ( s ) )

Original: Since[ ρ − 1 ] ii ≥ 0 , g (1) i ≤ g (0) i .
Revised: Since [ ρ − 1 ] ii ≥ 0 , g (1) i ≤ g (0) i .

Original: Averaging the factor in parentheses in Eq.
Revised: Averaging the factor in parentheses in Eq.

Original: 7 over neurons , we additionallydefine ¯ g (1) i = µω i (cid:16) 1 − 1 N (cid:80) j ( µ CV 2 j ) − 1 [ ρ − 1 ] jj (cid:17) , which yields uniform mean rates across neuronsand, as long as (cid:80) j ( µ CV 2 j ) − 1 [ ρ − 1 ] jj is constant across environments, homeostasis as defined above.
Revised: 6 over neurons, we additionally define ¯ g (1) i = µω i (cid:16) 1 − 1 N (cid:80) j ( µ CV 2 j ) − 1 [ ρ − 1 ] jj (cid:17) , which yields uniform mean rates across neurons and, as long as (cid:80) j ( µ CV 2 j ) − 1 [ ρ − 1 ] jj is constant across environments, homeostasis of firing rates.

Original:  Validation of results
Revised:  Validation of results

Original: In our first-order expansion above, we used the condition (6).
Revised: In our first-order expansion above, we used the condition (5).

Original: How likely is this to hold in the cortex?
Revised: How likely is this to hold in the cortex?

Original:  We see from the solution that µ scales the average spike count of the neurons over the rate coding time-interval.
Revised: • We see from the solution that µ scales the average spike count of the neurons over the rate coding time-interval.

Original: Typical cortical firing rates are 10 Hz [1], so if neurons are codingover an interval of 0.1 sec.
Revised: Typical cortical firing rates are 10 Hz [1], so assuming a coding interval of 0.1 sec.

Original: , we obtain µ ≈ 1 .
Revised: , we obtain µ ≈ 1 .

Original: If we take our model neurons to represent a cluster of m cortical neurons with similar tuning, µ can be further scaled up by m 
Revised: Taking our model neurons to represent a cluster of m cortical neurons with identical tuning, µ is further scaled up by m .

Original: • ρ is the signal correlation matrix of neurons.
Revised: • ρ is the signal correlation matrix of neurons.

Original: Stringer et al.
Revised: Stringer et al.

Original: [9] have found that in  the 
Revised: [9] have found that in V1, the eigenvalues

Original: primary visual cortex (V1), we can expect the eigenvalues of ρto obey a power law decay, approximatelyas 1 /n .
Revised: of ρ obey a power law decay of approximately 1 /n .

Original: This leads to an average value of [ ρ − 1 ] ii of approximately ln( N ) / 2 .
Revised: This leads to an average the average of [ ρ − 1 ] ii of approximately ln( N ) / 2 .

Original: Even forvery large values of N = O (10 8 ) this remains bounded above by
Revised: Even for large values of N = O (10 8 ) this remains bounded above by

Original: • The coefficient of variation CV can be seen as a measure ofcoding level or sparseness ofneural responses.
Revised: • The coefficient of variation CV can be seen as a measure of sparseness of responses.

Original: Consider the toy-model Ω i ( s ) ∼ Bern( p i ) (for an Ω i ( s ) with two outputlevels: 0 and 1).
Revised: Consider the toy-model Ω i ( s ) ∼ Bern( p i ) (for an Ω i ( s ) with two output levels: 0 and 1).

Original: In this case, CV 2 i = (1 − p i ) /p
Revised: In this case, CV 2 i = (1 − p i ) /p

Original: i , which is large for small p i .
Revised: i , which is large for small p i .

Original: At p i = 0 .
Revised: At p i = 0 .

Original: 1we obtain CV 2 i = 9 .
Revised: 1 we obtain CV 2 i = 9 .

Original: Values of p i in this range have been observed in empirical studies [1]. 
Revised: Values of p i in this range have been observed in empirical studies [1]. This makes it

Original: This makes it clear when we should expect homeostasis – when firing rates are high but selective,and signal correlation structure corresponds to a high-dimensional geometry, as e.g.observed in [9].
Revised: clear when we should expect homeostasis – when firing rates are high but selective, and signal correlation structure corresponds to a high-dimensional geometry, as e.g.

Revised: observed

Original:  When these conditions are violated, homeostasis deviates from optimally.
Revised: in When these conditions are violated, homeostasis deviates from optimally.

Original: We numerically compare the performance of the zeroth-order homeostatic codewith gains g (0) i =
Revised: We numerically compare the performance of the zeroth-order homeostatic code g (0) i =

Original: µ/ω i , the first order correction, g (1) i , and the homeostatic approximation to the latter, ¯ g (1) i , againstnumerically optimised gains, g opti , on the objective L ( g )  .
Revised: µ/ω i , the first order correction, g (1) i , and the homeostatic approximation to the latter, ¯ g (1) i , against gains g opti which have been numerically optimised by performing simple gradient ascent on the objective L ( g ) , initialised as g (1) i .

Original: An environment(originally specified bythe stimulus Z ( s ) density) is specified by ρ , ω i , and CV i .
Revised: An environment is specified by ρ , ω i , and CV i (via Z ( S ) ).

Original: We consider a sequence of environmentsparameterised by ϵ ∈ [0 , 1] .
Revised: We consider a sequence of environments parameterised by ϵ ∈ [0 , 1] .

Original: In environment ϵ , ω j ( ϵ ) = 4 − ϵ cos(2 πj/N ) ; this can be thought of asan environmental shift in which some stimuli become up to 25% more or less prevalent.
Revised: In environment ϵ , ω j ( ϵ ) = 4 − ϵ cos(2 πj/N ) ; this can be thought of as an environmental shift in which some stimuli become up to 25% more or less prevalent.

Original: We took {+N = 100 , µ = 10 , and CV i ( ϵ ) = 
+}
Revised: We took {+N = 100 , µ = 10 , and CV i ( ϵ ) = 
+}

Original: N = 100 , µ = 10 , and CV i ( ϵ ) = 3 for all environments and neurons.ρ ( ϵ ) is obtained by normalising a positive-definite  matrix Σ ϵ  which has a 1 /n eigen-spectrum, in linewith the findings of [9].
Revised: for all environments and neurons. ρ ( ϵ ) is obtained by normalising a positive-definite covariance matrix Σ( ϵ ) which has a 1 /n eigenspectrum, in line with the findings of [9].

Original: The  eigen-bases of the end points Σ 0 and Σ 1 are sampled randomly andindependently, and we smoothly interpolate between these to obtain the eigen-bases for Σ ϵ  , ϵ ∈ (0 , 1) .(see
Revised: The orthonormal eigen-bases of the end points Σ(0) and Σ(1) are sampled randomly and independently, and we smoothly interpolate between these to obtain the eigen-bases for Σ( ϵ ) , ϵ ∈ (0 , 1) .

Original:  App.
Revised: (see App.

Original: A.3 for details).
Revised: A.3 for details).

Original: To compare the performance of each approximate solution, g app ( ϵ ) , we use the measure
Revised: To compare the performance of each approximate solution, g app ( ϵ ) , we use the measure

Original: [Equation]
Revised: [Equation]

Original: which can be interpreted as the improvement in L ( · ; ϵ ) achieved by the adaptive g app ( ϵ ) over theunadapted optimal gains in the original environment g opt (0) , relative to the same improvementobtained by the optimally adaptive gains.
Revised: which can be interpreted as the improvement in L ( · ; ϵ ) achieved by the adaptive g app ( ϵ ) over the unadapted optimal gains in the original environment g opt (0) , relative to the same improvement obtained by the optimally adaptive gains.

Original: C ( ϵ ) for different approximations are plotted in Fig.
Revised: C ( ϵ ) for different approximations are plotted in Fig.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: ω

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: (cid:15) =

Original: [Figure]
Revised: [Figure]

Original: We first note that the first order approximation (7) performs essentially as well as numerical optimisation of L .
Revised: We first note that the first order approximation (6) performs essentially as well as numerical optimisation of L .

Original: The homeostatic correction ¯ g (1) ( ϵ ) also performs close to optimally for ϵ ≥ 0 .
Revised: The homeostatic correction ¯ g (1) ( ϵ ) also performs close to optimally for ϵ ≥ 0 .

Original: Moreover,as soon as ϵ exceeds about 0.2, exact homeostasis g (0) is superior to no adaptation.
Revised: Moreover, as soon as ϵ exceeds about 0.2, exact homeostasis g (0) is superior to no adaptation.

Original: For very smallchanges in the sensory environment, exact homeostasis performs worse than the (constant) numerically optimised code for ϵ = 0 , because g (0) lacks knowledge of the signal correlation structureof responses.

Original: Finally, we also computed the mean relative errors 1 N (cid:80)
Revised: Finally, we also computed the mean relative errors 1 N (cid:80)

Original: j | g appj − g optj | /g optj of theapproximate solutions.
Revised: j | g appj − g optj | /g optj of the approximate solutions.

Original: These were approximately constant in ϵ to 3 significant figures, and took thevalues 0 . 0294 , 0 . 000291 , 0 . 00615 , for g (0) , g (1) , and ¯ g (1) , respectively.
Revised: These were approximately constant in ϵ to 3 significant figures, and took the values 0 . 0294 , 0 . 000291 , 0 . 00615 , for g (0) , g (1) , and ¯ g (1) , respectively.

Original:  Bayes-ratio coding
Revised:  Bayes-ratio coding

Original: In this section, we neglect correction terms and work with the homeostatic approximation g (0)
Revised: In this section, we neglect correction terms and work with the homeostatic approximation g (0)

Original: i =
Revised: i =

Original: µ/ω i , which yields h i ( s ) = µ Ω i ( s ) / E
Revised: µ/ω i , which yields h i ( s ) = µ Ω i ( s ) / E

Original: [Ω i ( S )] .
Revised: [Ω i ( S )] .

Original: So far we have been silent on the computationencoded by the representational curves Ω ( s ) .
Revised: So far we have been silent on the computation encoded by the representational curves Ω ( s ) .

Original: Here, we apply the homeostatic code to a form of
Revised: Here, we apply the homeostatic code to a form of

Original: Bayesian encoding, the distributed distributional code (DDC) [11].
Revised: Bayesian encoding, the distributed distributional code (DDC) [11].

Original: In a DDC, which is based onan internal generative model of stimuli with latent variables z , Ω i ( s ) is the posterior expectation(given the observed s ) of a so-called kernel function k i ( z ) .
Revised: In a DDC, which is based on an internal generative model of stimuli with latent variables z , Ω i ( s ) is the posterior expectation (given the observed s ) of a so-called kernel function k i ( z ) .

Original: We make an ideal-observer assumption,wherein the internal generative model generates the true stimulus distribution, Z ( s ) .
Revised: We make an ideal-observer assumption, wherein the internal generative model generates the true stimulus distribution, Z ( s ) .

Original: In this case,the homeostatic adaptation yields h i ( s ) =
Revised: In this case, the homeostatic adaptation yields h i ( s ) =

Original: µ E [ k i ( z ) | s ] E
Revised: µ E [ k i ( z ) | s ] E

Original: [ k i ( z )] .
Revised: [ k i ( z )] .

Original: For the special case k i ( z ) =
Revised: For the special case k i ( z ) =

Original: δ ( z i − z ) , thisyields Bayes-ratio coding :
Revised: δ ( z i − z ) , this yields Bayes-ratio coding :

Original: [Equation]
Revised: [Equation]

Original: where Π and π denote the posterior and prior distributions over z , and f ( s | z ) is the generative model’slikelihood.
Revised: where Π and π denote the posterior and prior distributions over z , and f ( s | z ) is the generative model’s likelihood.

Original: As shown in App.
Revised: As shown in App.

Original: A.4, Bayes-ratio coding can be achieved via divisive normalisation, acanonical cortical operation [3], with adaptive normalisation weights proportional to the prior π ( z i )(allowed to vary across environments) and feedforward inputs given by the likelihood function.
Revised: A.4, Bayes-ratio coding can be achieved via divisive normalisation, a canonical cortical operation [3], with adaptive normalisation weights proportional to the prior π ( z i ) (allowed to vary across environments) and feedforward inputs given by the likelihood function.

Original: Additionally, Bayes-ratio coding can be used to explain certain stimulus-specific adaptation effects.
Revised: Additionally, Bayes-ratio coding can be used to explain certain stimulus-specific adaptation effects.

Original: Itis typical of stimulus specific adaptation in V1 [2] that orientation tuning curves display a repulsionand suppression around the over-represented orientation of the adaptor stimulus.
Revised: It is typical of stimulus specific adaptation in V1 [2] that orientation tuning curves display a repulsion and suppression around the over-represented orientation of the adaptor stimulus.

Original: To model the findingsof [2] with Bayes-ratio coding, we took stimulus and latent variable spaces to be the orientationspace [ − 90 , 90) , and the likelihood f
Revised: To model the findings of [2] with Bayes-ratio coding, we took stimulus and latent variable spaces to be the orientation space [ − 90 , 90) , and the likelihood f

Original: ( s | z i ) =
Revised: ( s | z i ) =

Original: ψ ( s − z i ; 10) where ψ ( · ; k )is a von Mises densitywith precision k , with z i  uniformly spaced over [ − 90 , 90) .
Revised: ψ ( s − z i ; 10) where ψ ( · ; k ) a von Mises density with precision k , where z i are uniformly spaced over [ − 90 , 90) .

Original: We consider a uniform pre-adaptationprior π pre , and a post-adaptation prior π post ( z ) = 0 .
Revised: We consider a uniform pre-adaptation prior π pre , and a post-adaptation prior π post ( z ) = 0 .

Original: 6 180 + 0 .
Revised: 6 180 + 0 .

Original: 4 ψ ( z ; 2) which represents an adaptorstimulus with orientation z = 0 ◦ .
Revised: 4 ψ ( z ; 2) representing an adaptor stimulus at z = 0 ◦ .

Original: The pre- and post-adaptation tuning curves are plotted in figure 3,and exhibit the suppression and repulsion of the adapted tuning curves around the over-representedstimulus.
Revised: The pre- and post-adaptation tuning curves are plotted in figure 2, and exhibit the suppression and repulsion of the adapted tuning curves around the over-represented stimulus.

Original: This result generalises to a homeostatic DDC with finite-width unimodal kernel functions.
Revised: This result generalises to a homeostatic DDC with finite-width unimodal kernel functions.

Original: [Figure]
Revised: [Figure]

Original:  A Appendix
Revised:  Conclusion

Revised: We developed a theory of optimal gain modulation for combatting noise in neural representations. We demonstrated that, when mean neural firing rates are not too small and responses are sufficiently sparse with a high-dimensional geometry, the trade-off between coding fidelity and metabolic cost is optimised by gains that react to shifts in environmental stimulus statistics to yield firing rate homeostasis. Lastly, we demonstrated that, when applied to Bayesian DDCs as an example of neural representation, homeostasis leads to characteristic suppression and repulsion of tuning curves as observed in V1.

Revised: In future research, we will extend our analysis to the more realistic case of Poisson spiking noise ( i.e. n i | S ∼ Poisson( h i ( S )) ), and investigate the extent to which homeostasis deviates from optimally as the condition µ CV i ≫ [ ρ − 1 ] ii ceases to hold.

Original:  A.1 Upper bound on L0
Revised: A Appendix A.1 Upper bound on L

Original: Recall that n i | s ∼ N
Revised: Recall that n i | s ∼ N

Original: ( h i ( s ) , h i ( s )) .
Revised: ( h i ( s ) , h i ( s )) .

Original: In this case,
Revised: In this case,

Original: [Equation]
Revised: [Equation]

Original: This gives us equation (1).

Original: We now use the fact that the entropy of any continuous distribution canbe upper bounded by the entropy of a Gaussian with with the same covariance.
Revised: We now use the fact that the entropy of any continuous distribution can be upper bounded by the entropy of a Gaussian with with the same covariance.

Original: The entropy of a N ( µ , Σ) random variable is
Revised: The entropy of a N ( µ , Σ) random variable is

Original: [Equation]
Revised: H [ N ( µ , Σ)] = N 2 ln(2 πe ) + 12 ln(det(Σ)) .

Original: It therefore suffices to find the covariance of n .
Revised: It therefore suffices to find the covariance of n .

Original: We use the decomposition Cov( n ) =
Revised: We use the decomposition Cov( n ) =

Original: E [Cov( n | s )]
Revised: E [Cov( n | s )]

Original: + Cov( E [ n | s ]) and compute
Revised: + Cov( E [ n | s ]) and compute

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Cov( Ω ( s ))diag( g ) = diag( g )diag( ω ) [ I + diag( ω ) − 1 Cov(Ω( s ))diag( ω ) − 1 diag( ω )diag( g )] = diag( g)diag( ω )
Revised: g

Original: [ I + P diag( ω ⊙ g )] ln(det(Cov( n ))) =
Revised: [Equation]

Original: ln(det(diag( g ))) + ln(det(diag( ω ))) + ln(det( I + P diag( ω ⊙ g ))) = N (cid:88) i
Revised: [Equation]

Original: = ln ( g i ) + N (cid:88) i = ln( ω i ) + ln (det ( I + P diag( ω ⊙ g ))) where P = diag( ω ) − 1 Cov( Ω ( s ))diag( ω ) − 1 .
Revised: where P = diag( ω ) − 1 Cov( Ω ( s ))diag( ω ) − 1 .

Original: Putting this together, we obtain:
Revised: Putting this together, we obtain:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: Finally, substituting Eqs.

Original: We now insert the above expressions to obtain the  bound on mutual information:
Revised: (9) and (10 in the definition for the mutual information, we obtain the upper bound I

Original: [Equation]
Revised: [Equation]

Original: or
Revised: n , [Equation] [Equation] [Equation]

Revised: H [ n |

Original: [Equation]
Revised: [Equation]

Revised: or 2 I ( n , s ) ≤ ln (det ( I + P diag( ω ⊙ g )))

Revised: + const.

Revised: in g .

Original: A.2 Approximate maximisation of L
Revised: A.2 Approximate maximisation of L

Original: We now consider optimising
Revised: We now consider optimising

Original: [Equation]
Revised: [Equation]

Original: Taking the derivative with respect to g i , we obtain
Revised: Taking the derivative with respect to g i , we obtain

Original: [Equation]
Revised: [Equation]

Original: To obtain an approximate solution for this we use a first order Neumann expansion for ( I +( P diag( ω ⊙ g )) − 1 ) − 1 .
Revised: To obtain an approximate solution for this we use a first order Neumann expansion for ( I + ( P diag( ω ⊙ g )) − 1 ) − 1 .

Original: This requires that ( P diag( ω ⊙ g )) − 1 has a small norm, i.e. P diag( ω ⊙ g )is large in norm.
Revised: This requires that ( P diag( ω ⊙ g )) − 1 has a small norm, i.e. P diag( ω ⊙ g ) is large in norm.

Original: The Neumann expansion gives:
Revised: The Neumann expansion gives:

Original: [Equation]
Revised: [Equation]

Original: Plugging this in, we obtain
Revised: Plugging this in, we obtain

Original: [Equation]
Revised: [Equation]

Original: Setting equation (9) to zero, and solving to find the approximate maximiser of L ( g ) , we get:
Revised: Setting equation (11) to zero, and solving to find the approximate maximiser of L ( g ) , we get:

Original: [Equation]
Revised: [Equation]

Original: We further approximate by taking a first order Taylor expansion of the square root to obtain equation(7):
Revised: We further approximate by taking a first order Taylor expansion of the square root to obtain equation (6):

Original: [Equation]
Revised: [Equation]

Original: where ρ is the correlation matrix Corr( Ω ( s )) and CV i is the coefficient of variation, 
Revised: where ρ is the correlation matrix Corr( Ω ( s )) and CV i is the coefficient of variation, √ [Equation] s )

Original: [Equation]
Revised: [Equation]

Original:  The expansions employed in this argument require that ( P diag( ω ⊙ g )) − 1 is small in norm, and thatρ − 1 ii ≪ µ CV 2 i .
Revised: ω [Equation] The expansions employed in this argument require that ( P diag( ω ⊙ g )) − 1 is small in norm, and that ρ − 1 ii ≪ µ CV 2 i .

Original: Note that ( P diag( ω ⊙ g )) ij =
Revised: Note that ( P diag( ω ⊙ g )) ij =

Original: CV
Revised: CV

Original: i ρ ij CV j g j ω j .
Revised: i ρ ij CV j g j ω j .

Original: But we know that g
Revised: But we know that g

Original: j
Revised: j

Original: =
Revised: =

Original: O
Revised: O

Original: µ/ω j )and therefore g j ω j
Revised: µ/ω j ) and therefore g j ω j

Original: = O ( µ ) .
Revised: = O ( µ ) .

Original: Accordingly, we obtain that ( P diag( ω ⊙ g )) − 

Revised: Accordingly, we obtain that ( P diag( ω ⊙ g )) −

Original: =
Revised: =

Original: O (cid:16) ρ − 1 µ CV 2 (cid:17) .
Revised: O (cid:16) ρ − 1 µ CV 2 (cid:17) .

Original: Thismeans that the condition for truncating the Neumann expansion is indeed Eq.
Revised: This means that the condition for truncating the Neumann expansion is indeed Eq.

Original: A.3 Specification of environment for ϵ
Revised: A.3 Specification of environment for ϵ

Original: To generate ρ ( ϵ ) , we first generate a covariance matrix Σ ϵ  , and let ρ ( ϵ ) be the correspondingcorrelation matrix.
Revised: To generate ρ ( ϵ ) , we first generate a covariance matrix Σ( ϵ ) , and let ρ ( ϵ ) be the corresponding correlation matrix.

Original: The procedure for generating Σ ϵ  is as follows.
Revised: The procedure for generating Σ( ϵ ) is as follows.

Original: We randomly and independently sample two N × N random Gaussian matrices
Revised: We randomly and independently sample two N × N random Gaussian matrices

Original: A 0 , A 1 ∼
Revised: A 0 , A 1 ∼

Original: N N × N (0 , I N × N ) and obtain symmetric matrices S 

Revised: N N × N (0 , I N × N ) and obtain symmetric matrices S

Original: =
Revised: =

Original: A 

Revised: A

Original: + A T 0 and S
Revised: + A T 0 and S

Original: 1 =
Revised: 1 =

Original: A 1 + A T 1 .
Revised: A 1 + A T 1 .

Original: As iswell known, the eigen-basis (represented by an orthogonal matrix) of such a random Gaussian matrixis distributed uniformly ( i.e. , according to the corresponding Haar measure) over the orthogonalgroup O ( N ) .
Revised: As is well known, the eigen-basis (represented by an orthogonal matrix) of such a random Gaussian matrix is distributed uniformly ( i.e. , according to the corresponding Haar measure) over the orthogonal group O ( N ) .

Original: Now let S  ϵ 
Revised: Now let S ( ϵ ) =

Original: = (1 − ϵ ) S 0 + ϵS 1 .
Revised: (1 − ϵ ) S 0 + ϵS 1 .

Original: Almost surely these matrices have a non-degenerate spectrum,and hence a unique representation as S  ϵ 
Revised: Almost surely these matrices have a non-degenerate spectrum, and hence a unique representation as S ( ϵ ) =

Original: = U ( ϵ )Λ( ϵ ) U
Revised: U ( ϵ )Λ( ϵ ) U

Original: ( ϵ ) T where Λ( ϵ ) is diagonal with strictlydecreasing eigenvalues and U ( ϵ ) is orthogonal.
Revised: ( ϵ ) T where Λ( ϵ ) is diagonal with strictly decreasing eigenvalues and U ( ϵ ) is orthogonal.

Original: Moreover, U ( ϵ ) depends continuously on ϵ .
Revised: Moreover, U ( ϵ ) depends continuously on ϵ .

Original: Let D =
Revised: Finally, we define Σ( ϵ ) = U ( ϵ ) DU ( ϵ ) T , where D =

Original: diag(1 , 1 / 2 , 1 / 3 , . .
Revised: diag(1 , 1 / 2 , 1 / 3 , . .

Original: , 1 /N ) .
Revised: , 1 /N ) .

Original: Then we define Σ ϵ =

Original: U ( ϵ ) DU ( ϵ ) T .

Original: A.4 Bayes-ratio coding and divisive normalisation
Revised: A.4 Bayes-ratio coding and divisive normalisation

Original: Given a collection of feed-forward inputs, F i ( s ) , divisive normalisation computes the response (andthus the tuning curve) of neuron i as
Revised: Given a collection of feed-forward inputs, F i ( s ) , divisive normalisation computes the response (and thus the tuning curve) of neuron i as

Original: [Equation]
Revised: [Equation]

Original: (11) where w j are a collection of normalisation weights , and γ, σ ≥ 0 and n ≥ 1 are constants.
Revised: where w j are a collection of normalisation weights , and γ, σ ≥ 0 and n ≥ 1 are constants.

Original: Bayes-ratio coding can be achieved naturally by a divisive normalisation model in which n = 1 andthe feed-forward inputs are given by the generative model’s likelihood function f
Revised: Bayes-ratio coding can be achieved naturally by a divisive normalisation model in which n = 1 and the feed-forward inputs are given by the generative model’s likelihood function f

Original: ( s | z i ) .
Revised: ( s | z i ) .

Original: We thenchoose the normalisation weights w i to encode prior probabilities, w i =
Revised: We then choose the normalisation weights w i to encode prior probabilities, w i =

Original: π ( z i )
Revised: π ( z i )

Original: δ z i , where the volumeelement δ z i is chosen such that the latent variable space is the disjoint union of volumes of size δ z ieach containing their corresponding sample point z i .
Revised: δ z i , where the volume element δ z i is chosen such that the latent variable space is the disjoint union of volumes of size δ z i each containing their corresponding sample point z i .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: hence
Revised: hence

Original: [Equation]
Revised: [Equation]

Original: Taking the limit as σ → 0 , we obtain Eq.8:
Revised: Taking the limit as σ → 0 , we obtain Eq.

Original: [Equation]
Revised: [Equation]

Original: Therefore, provided σ is small compared to Z ( s ) , divisive normalisation can be used to approximate
Revised: Therefore, provided σ is small compared to Z ( s ) , divisive normalisation can be used to approximate

Original: Bayes-ratio coding.
Revised: Bayes-ratio coding.

Original: Not only does this show that implementing Bayes-ratio coding is biologicallyplausible, this framework gives a normative interpretation to both the feedforward inputs (as thegenerative model likelihoods) and the normalisation weights (as the prior probabilities).
Revised: Not only does this show that implementing Bayes-ratio coding is biologically plausible, this framework gives a normative interpretation to both the feedforward inputs (as the generative model likelihoods) and the normalisation weights (as the prior probabilities).