Biologically plausible solutions for spiking networks with efficient coding
AffiliationAddress
Abstract
Understanding how dynamics of neural networks is shaped by the computations thenetworks perform is a fundamental topic in neuroscience.
Recently, the frameworkof efficient coding proposed a theory on how spiking neural networks can computelow-dimensional stimulus signals with high efficiency.
Efficient spiking networksare based on time-dependent minimization of a cost function related to informationcoding with spikes.
To inform the understanding of the function and dynamics ofbiological networks in the brain, however, the mathematical models have to obeyconstraints of biological networks.
Currently, spiking network models of efficientcoding have been extended to include some features of biological plausibility, suchas the inclusion of architectures with excitatory and inhibitory neurons.
However,biological realism of efficient coding theories is still limited to simple cases anddoes not include single neuron and network properties that are known to be important in biological circuits.
Here, we revisit the theory of efficient coding withspikes to develop spiking neural networks that are closer to biological circuits.
Namely, we find a biologically plausible spiking model realizing efficient codingin the case of a generalized leaky integrate-and-fire network with excitatory andinhibitory units, equipped with fast and slow synaptic currents, local homeostaticcurrents such as spike-triggered adaptation, hyperpolarization-activated reboundcurrent, heterogeneous firing thresholds and resets, heterogeneous postsynapticpotentials, and structured, low-rank connectivity.
We show how the complexity of E-E connectivity matrix shapes network responses.
Introduction
Complex biological systems are thought to be shaped through evolution to optimally perform a setof desired functions within the constraints of biological building blocks they are made of.
A bulkof previous influential work [30, 21] has proposed that the organization of the mammalian cortex,the most sophisticated biological organ known to humanity, makes no exception and follows too thegeneral principles of evolution towards optimality under biological constraints.
One prominent theory is that the function of the cerebral cortex is organized for efficient coding, inthat it optimizes the encoding of information under a number of biological constraints such as themetabolic expenditure, the speed and the timing of information coding.
Imposed constraints can beformalized with a cost function.
To understand biological neural networks that operate in animal’s brains, the mathematical descriptionof networks optimizing functionally relevant cost functions has to include as many features of biological realism as possible, so that the constraints satisfied by the model network will be informative ofthose obeyed by biological neural circuits.
Thus, all elements of the model should have a counterpart in biological circuits [13] and be directly relatable to them.
It is still unclear how to derive efficientcoding theories of biologically plausible neuronal networks.
An influential proposal of efficient coding in cortical circuits has been on extracting statisticalregularities from static natural images with a sparse set of basis functions, which yielded basisfunctions that resemble receptive fields in V1 [30].
A related proposal in the domain of auditorystimuli yielded tuning properties of auditory nerve fibers [21].
In these studies, the goal of efficientcoding is to find sparse spatial or temporal filters that enables optimal information transmission aboutthe stimulus by the activity of each single neuron.
Analyses of cortical networks, however, show thatinformation processing tends to be distributed across neurons [15] and neglecting neuron-to-neuroninteractions typically reduces the information encoded in the spiking activity of neural circuits [37] orthe ability of use this information to produce relevant behavioral outputs [38, 43].
Moreover, analysesof local neural circuits show that millisecond precision of spike timing of sensory neurons carriesinformation about both static [34, 31] and dynamic stimuli [16, 28], which suggests that the timingof spiking activity is highly relevant for the information processing in the cortex.
Analyses of neuraldata suggest that meaningful theories of coding in the real cerebral cortex must rely on spiking neuronmodels and on population codes.
To address this issue, models of how to optimally perform a given coding function have been extendedto spiking dynamics.
The cost function includes a quadratic error between the desired signal anda linear read-out of population activity and the distance between the signal and the read-out isminimized with every spike [1, 4, 2].
However, the efficient coding with spikes, and the related theoryof predictive coding [46], have been so far developed as abstract mathematical frameworks that lackbiological realism.
Namely, these models are not consistent with the observation that a single neuroncan either be excitatory or inhibitory, but not both (Dale’s law).
Very recently, progress has beenmade in this direction by deriving efficient coding with spiking networks with a more biologicallyplausible neural circuit architecture with interacting excitatory (E) and inhibitory (I) neurons [22],opening the possibility to model neural computations with biologically relevant models.
Here, we make major further progress by deriving a mathematically well defined extension of efficientcoding for considerably more biologically plausible spiking neural network models of cortical circuits.
We develop a recurrently connected neural circuit with E and I cell type that performs a genericlinear transformation between its inputs and its outputs.
We impose the E-I architecture and recover anetwork with low-rank and structured connectivity that maximizes the information transmission aboutthe stimulus with every spike.
Our proposed solution recovers a generalized leaky integrate-and-fire(LIF) neuron model under the constraints of Dale’s law.
Our solution also takes into account fastermembrane time constant of I compared to E neurons, compatible with the biophysics of pyramidalneurons and fast-spiking inhibitory interneurons.
Our derivations prescribe subthreshold dynamics,firing thresholds and resets after spiking, thus giving a complete description of a spiking neuralnetwork along with network’s computation.
Importantly, all elements of the model are biologicallyplausible and directly relatable to measurable currents in biological networks.
We start by introducing the cost functions and explaining how the spike timing is conditioned on aminimization of cost functions of E and I neurons in every time step.
We continue with an outline ofthe analytical treatment of cost functions that leads to expressions of membrane potentials and firingthresholds of a generalized LIF network model with structured, low-rank connectivity.
Further on,we provide a description and biologically plausible interpretation of derived currents.
Finally, wesimulate the network with biologically plausible parameters and analyze the behavior of the networkwith respect to the complexity of E-E synaptic connections.
Results
2.1 Minimization of the loss function with spikes
From a spiking network of N E excitatory and N I inhibitory neurons, we define a linear populationread-out of the spiking activity of E and I neurons: [Equation] [Equation]
[Equation]
with λ Y > 0 the inverse time constant of the read-out.
In the population read-out, the decodingmatrix W Y weights the vector of spikes f Y ( t ) =
[ f Y 1 ( t ) , . .
f YN Y ( t )] ⊺ .
The M -by- N Y matrix W Yassociates each of the N E excitatory ( N I inhibitory) neurons with M weights, W Y = ( w Ymi ); m =
, M, i = 1 , . . .
, N Y , where N Y is the number of neurons of the cell type Y , and M is thenumber of stimulus features estimated by the network.
The population read-out in eq.
(1) is anestimate of the desired signal x ( t ) , that is an abstract signal, unknown to the network.
The desired signal x ( t ) is a linear function of stimulus features s ( t ) =[
s 1 ( t ) , . .
, s m ( t ) , . . .
, s M ( t )] ⊺ and is defined as follows:
[Equation]
where A = ( a mn ); m, n = 1 , ..., M is a square matrix determining the linear transformation betweenthe input features s ( t ) and the desired signal x ( t ) .
We will refer to the transformation betweenthe features and the desired signal as network’s computation.
If A is a diagonal matrix, network’scomputation operates independently across features, while a non-diagonal matrix A implementslinear mixing of features.
We pose that the objective of the excitatory network is to minimize the distance between the desiredsignal and the population read-out of E neurons, while the objective of the inhibitory population is tominimizes the distance between the read-out of the E and I populations [2, 6].
Besides the codingerror, we also take into account the metabolic cost on spiking, since spiking activity is energeticallyexpensive for animal agents [29].
These objectives are formalized by loss functions with a quadraticcoding error and a quadratic regularizer, relative to E and I cell type:
[Equation]
(3b) where constants µ Y > 0 are weighting the relative contribution of the metabolic cost over the codingerror.
An alternative to the cost function in eq.
(3b) would be to minimize the distance between theread-out of I population and the signal, however, such a formulation, without further approximations,leads to a network where E and I neurons are unconnected.
The variable r Yi ( t ) is defined as alow-pass filtered spike train of the neuron i of cell type Y ∈ { E, I } ,
[Equation]
(4) with α Yi > 0 the inverse time constant of the low-pass filter.
As we gather the variable r Yi ( t ) acrossneurons, we get a vector of time-dependent neural activities r Y ( t ) =
[ r Y 1 ( t ) , . . .
, r YN Y ] ⊺ .
We pose that there will be a spike of neuron i of cell type Y at time t = t + if and only if this decreasesthe cost function, i.e. [Equation]
[Equation]
where the noise term η Yi ( t ) is added to account for randomness of the spike generation in biologicalneurons.
The noise term η Yi ( t ) =
σ Yi ξ Yi ( t ) is defined as a Gaussian random variable ξ Yi ( t ) with zeromean and covariance ⟨ ξ i ( t ) ξ j ( t ′ ) ⟩ = δ ij δ ( t − t ′ ) , and σ Yi is the noise intensity.
Contrary to previousapproaches where the noise is added in the membrane potential [2, 19], we here assume noise inthe condition for spiking.
While the noise in the membrane potential models synaptic inputs thatare unrelated to the coding function of the network, the noise in spike generation captures a sourceof noise that is intrinsic to neurons, namely the noise in the fraction of open ionic channels for agiven membrane potential [9].
Opening and closing of sodium and potassium channels is responsiblefor the generation of spikes, but the fraction of open channels for a given membrane potential isprobabilistic.
Assuming that a spike is only fired if this decreases the cost function as in eq.
(5), we apply ananalytical treatment of eqs.
3a-3b in parts similar to [2], and obtain the following expressions:
[Equation]
where vectors of firing thresholds, θ Y =
[ θ Y 1 , . . .
, θ YN Y ] ⊺ for Y ∈
{ E, I } , have elements
[Equation]
with ∥ w Yi ∥ 22 the squared length of decoding vector of the local neuron, ∥ w Yi ∥ 22 = (cid:80) Mm =1 ( w Ymi ) 2 .
Meanwhile, η Y ( t ) is the vector of noise processes, η Y ( t ) =
[ σ Y 1 ξ Y 1 ( t ) , . . .
, σ YN Y ξ YN Y ( t )] .
While theright-hand side of inequalities in (6a) and (6b) are firing thresholds, we assume that the left-handsides are proportional to the membrane potentials of E and I neurons, u E ( t ) and u I ( t ) , respectively,with u Y ( t ) =
[ u Y 1 (1) , . . . , u YN Y ( t )] ⊺ .
To express the temporal dynamics of the membrane potentials, ˙ u E ( t ) and ˙ u I ( t ) , we take the timederivative of all time-dependent terms on the left-hand side of eqs.
(6a) and 6b.
Contrary to previousapproaches and without loss of generality, we express the mixing matrix A as a difference of a squarematrix B and a diagonal matrix λ
E I MxM ,
[Equation]
with I the identity matrix.
Seen that cost functions minimize the distance between the signal andthe E estimate (eq. 3a), and between the E and I estimates (eq. 3b), we use the approximationsx ( t )
≈ ˆ x E ( t ) and ˆ x E ( t )
≈
ˆ x I ( t ) . Moreover, we remove synaptic connections that violate Dale’s law(see the Supplementary material).
After re-arranging the terms and without further approximations,interestingly, the network model can be expressed as a generalized LIF model, i.e., the LIF modelwith spike-triggered adaptation [23].
2.2 Generalized integrate-and-fire network with low-rank structured connectivity
Due to aforementioned approximations, we write the membrane potential as u Yi ( t )
≈ C
m V Yi ( t ) ,with C m the capacitance of the cell membrane.
The subthreshold dynamics of E and I neurons isthen dependent on the following currents:
[Equation]
and complemented with fire-and-reset rule: if V Yi ( t − ) ≥ ϑ Yi ( t − ) → V Yi ( t + ) =
V reset Y i .
Forsimplicity, we set the resting potential at 0, but a biologically plausible resting potential can beintroduced without affecting network’s dynamics.
From eqs.(6a), (6b) and (7), the effective thresholdthat takes into account the noise in the condition for spiking (eq. 5) is the following:
[Equation]
while the reset potentials for E and I cell types are
[Equation]
The firing thresholds in eq.
10 and reset potentials in eqs.
11a-11b are thus proportional to the lengthof decoding vector of the local neuron ∥ w Yi ∥ 2 , and to the regularizer µ Y that affects equally all theneurons from the same cell type.
If decoding weights w Ymi are heterogeneous across neurons, firingthresholds and resets are heterogeneous.
The noise at threshold in eq.
(10) implements the escapenoise that has been shown to be relevant for modeling of randomness in spike generation in corticalneurons [11].
The membrane equations in eqs.
9a-9b depend on a number of currents that, interestingly, all have astraightforward counterpart in biological networks.
The first currents on the right-hand site of eqs.9a-9b are leak currents,
[Equation]
[Equation]
that result from absorbing of several terms, among others the diagonal matrix λ
E I MxM from eq.
The feedforward current to E neurons is proportional to the weighted sum of external inputs s k ( t ) ,
[Equation]
Fast synaptic currents between E-to-I, I-to-I and I-to-E cells are given by a weighted sum of spikes ofpresynaptic neurons,
[Equation]
with structured connectivity matrices C Y Z ij , where the strength of synaptic connection is proportionalto the similarity of decoding vectors,
[Equation]
Note that synaptic connections between neurons with different selectivity (i.e., neuronal pairs withnegative dot product of decoding vectors, ( w Yi ) ⊺ w Zj < 0 ), have been set to zero to make the networkconsistent with Dale’s law (see the Supplementary material).
In particular, we had to ensure thata particular neuron can only send excitatory or inhibitory currents to other neurons, but not both.
Even though there is no direct fast synaptic connections between E-to-E neurons, effectively, fastsynaptic connectivity implements lateral inhibition between E neurons with similar selectivity.
Lateralinhibition (or competition) between E neurons with similar selectivity is a dynamical effect that isessential for an efficient neural code [17] and has been demonstrated in biological circuits [7].
Next, we have currents that are triggered by spiking of the local neuron,
[Equation]
The kinetics of the local current is determined by the time constant of low-pass filtered spike train,α Yi , while the sign and the strength of the current depend on the relation of time constants of thepopulation read-out λ Y , and the single neuron read-out α Yi .
To allow the population read-out ˆ x Y ( t )to track fast changes of the signal x ( t ) , we assume that the population read-out is fast, while thesingle neuron read-out, related to a homeostatic read-out of single neuron’s firing frequency [8], isassumed to be a slower process.
Such a relation of time constants, λ Y > α Yi , constrains the localcurrent in eq.
16 to spike-triggered adaptation.
If, on the contrary, λ Y < α Yi , we get spike-triggeredfacilitation, a current that might be relevant during learning and restructuring of synaptic connections[33].
Besides the difference of time constants between the population and single neuron read-out, thestrength of the local current depends on the constant of the regularizer µ Y .
The stronger the penaltyof the metabolic cost in the objective function in eq.
(3a) -(3b), the stronger the current in eq.
This makes most sense in case when the local current is spike-triggered adaptation, where a strongerregularizer increases the amplitude of the hyperpolarizing jump of the adaptation current and thusreduces the firing frequency of the spiking neuron.
Besides fast synaptic currents (eq. 14), there are slower synaptic currents between E-to-E and E-to-Ineurons: [Equation] (17a
[Equation]
(17b) with matrices of synaptic interactions:
[Equation]
where Y = E for E-to-E synapses and Y =
I for E-to-I synapses.
The synaptic time constant α Y i ineqs.
17a-17b is given by the time constant of low-pass filtered spike trains (eq. 4).
Same as with fastsynaptic currents in eq.
(14), the strength of the synapse between the presynaptic neuron j and thepostsynaptic neuron i in eq.
(18) depends on the similarity of decoding vectors of the presynapticand postsynaptic neurons, however, here the synaptic current depends also on the matrix B that wasexpressed from the mixing matrix
A (see eq.
To make slower synaptic currents consistent with
Dale’s law, we removed connections between neurons with different selectivity and constrained thematrix B to be positive semi-definite.
This ensures that presynaptic spikes of E neurons always causean excitatory current in the postsynaptic neurons.
The postsynaptic current in the neuron i causedby a spike of the presynaptic neuron j is: PSC slow EY ij =
D EYij exp (cid:0) − α Ei ( t − t ′ j,E ) (cid:1) with t ′ j,E thespike time of the excitatory presynaptic neuron j .
Note that with heterogeneous parameters α Yi , theduration of the PSC is heterogeneous across neurons.
Contrary to fast connections that implementlateral inhibition among E neurons with similar selectivity (see eq. 14), slower E-to-E synapticconnections implement cooperation among E units with similar selectivity.
E-to-E connections areimportant for the description of biological networks in the cortex [18].
Slower E-to-E synapses are complemented by slower E-to-I synapses (eq. 17b), where the excitatoryeffect of an E-to-I synapse constrains the following relation of time constants: λ I > λ E .
Since λ Eand λ I are membrane time constants of the E and I cell type, respectively (see eq. 12), this relationconstrains the membrane time constant in E neurons to be slower than in I neurons, consistent with
E cell type modeling pyramidal cells while the I cell type models fast-spiking interneurons [26].
Fast-spiking (somatostatin) interneurons are a prevalent class of inhibitory neurons in cortical circuits[42].
As pyramidal neurons account for about 80 % of cortical neurons, and fast-spiking interneuronsaccount for at least a half of inhibitory neurons in the cortex [44], our model altogether providesa description for about 90 % of cortical neurons.
Note that alternative solutions for slow currentsexist (see the Supplementary Material).
These alternative solutions are, however, of lesser biologicalrelevance since they either result in a network without E-to-E connections, or induce global imbalanceof E and I currents in the network.
The efficient E-I network has low-rank connectivity.
The rank of fast synaptic connectivity is, bydefinition of the connectivity matrices C (eq. 15) and D (eq. 18), equivalent or smaller than thenumber of features encoded by the network, M .
It is natural to assume that the number of activefeatures s m ( t ) is much smaller than the number of neurons in the network, M << N E , whichgives low-rank connectivity matrices.
Low-rank connectivity forces the neural activity to live on alow-dimensional manifold [25], and the relevance of the latter for the description of the dynamicsof biological neural ensembles has been strongly suggested by the data [10].
Note also that usingheterogeneous and sparse decoding weights w Yi gives heterogeneous and sparse synaptic connectivityin case of both fast and slower synapses.
Finally, we derived a depolarizing current triggered by the spike of the local excitatory neuron, that wetermed I rebound i ( t ) .
In the efficient spiking network, the diagonal of the E-to-E recurrent connectivitymatrix D EE (eq. 17a) implements a positive “self-connection" - as the neuron spikes, it generatesa local depolarizing current with slower kinetics of the low-pass filtered spike train.
We write thiseffect as follows:
[Equation]
Since the matrix B is constrained to be positive semi-definite (see eq. 18), the current in eq.
19is always depolarizing.
After a spike, the local neuron is reset to its reset potential, which, in Eneurons, is necessarily below the resting potential (see eq. 11a; the resting potential is here at 0mV).
Right after the spike, the local neuron is therefore strongly hyperpolarized, and precisely atthis moment, the rebound current activates.
The current I rebound i ( t ) thus creates a rebound of themembrane potential after a strong hyperpolarization.
In biology, a current with these properties isthe hyperpolarization-activated cation current (h-current [32]), a current with slower kinetics thatis important in the generation of network oscillations, spontaneous firing and in controlling theexcitability of cortical neurons [24, 36].
A schematic of the network is illustrated on Figure 1A.
2.3 Effect of complexity of E-E connectivity on network responses
In the following, we analyze the effect of the complexity of E-E connections on network responseswith analytical considerations and simulations.
We start by considering a simplified network modelwhere we assume the matrix A in eq.
(8) to be a diagonal matrix. A simple way to do so is to set theelements of the matrix B to 0, i.e., B = ( b mn ); b mn = 0 ∀ m, n = 1 , . . .
, M , which constrains thedesired signal to a leaky integration of input features that is independent across features,
[Equation]
The time constant of the desired signal x ( t ) is now equal to the time constant of the populationread-out of the E cell type eq. (1), and such a network is devoid of E-E (eq. 17a) and slower E-I (eq. 17b) synaptic currents, as well as of the rebound current (eq. 19).
We further simplify the networkby considering the special case where the time constants of the population and the single neuronread-out are equal, ( α Yi ) − 1 = ( λ Y ) − 1 ∀
i ; Y ∈ { E, I } , which sets the local currents in eq.(16) tozero.
To simulate such a simplified network, as well as more complex ones, we first have to determinedecoding weights w Ymi , since these are free parameters.
We use unstructured decoding weights thatwe draw from the normal distribution with zero mean and standard deviation σ Yw , w Ymi ∼ N (0 , σ Yw ) ,with σ Ew = 1 and the ratio σ Iw : σ Ew = 3 : 1 .
We chose decoding weights in the I cell type to bestronger than in the E cell type, but the number of I neurons is smaller then the number of E neurons,with a biologically plausible ratio N E : N I = 4 : 1 .
Decoding weights then determine fast synapticcurrents I EIi ( t ) , I IIi ( t ) and I IEi ( t ) through eqs.
14-15, as well as slower synaptic currents I slow EE i ( t )
and I slow IE i ( t ) through eqs.
17a-18.
As we stimulate the simplified network with a step input in one of the input features, the networkresponds with asynchronous spiking of neurons that are aligned to the active feature (Figure 1B).
Thefeedforward current to neuron i depends on the alignment of the input s ( t ) with neuron’s decodingvector w Ei (see eq. 13).
When a single feature is activated, e.g. s n ( t ) ̸ = 0 , a positive input, s n ( t ) > 0 ,drives neurons with positive decoding weight for the n − th feature (i.e., neurons with w Eni > 0 ),while a negative input, s n ( t ) < 0 , drives neurons with negative decoding weights (neurons withw Eni < 0 ).
If several features are active simultaneously, the feedforward current to E neurons isgiven by a linear mixture of features.
The simplified network is purely feedforward-driven andthe network response never outlasts the stimulus (Figure 1B).
The recurrent connectivity of such anetwork consists of structured fast connections between neurons with similar decoding selectivity(eq. 15), and the stronger the similarity of decoding vectors of neurons i and j , the stronger theirsynapse.
We now consider the network with E-E and slower E-I connections by choosing the matrix A to benon-diagonal.
The response of the network now critically depends on the strength and the structureof recurrent excitatory (E-E) connections.
The synaptic weight between excitatory neurons i andj depends not only on the similarity of decoding vectors w Ei and w Ej (as with fast connections),but also on the square matrix B (eq. 18).
Following the eq. 18, the matrix B has to be positivesemi-definite to ensure that the network obeys Dale’s law.
Positive semi-definiteness in turn constrainsthe matrix B to be symmetric and to have non-negative eigenvalues.
To ensure these properties, wewrite the matrix B as follows:
[Equation]
[Equation]
with a > 0 a positive constant influencing the strength of E-E connections and b 1 , . .
, b M ′ , a set oflinearly independent column vectors with entries b n =
[ b n 1 , . . . , b nM ] ⊺ , n = 1 , . .
, M ′ , and wherethe number of vectors is constrained as follows: 1 ≤ M ′ ≤ M . The matrix Γ determines the rankof the matrix B , as M ′ linearly independent column vectors set the rank of B to rank ( B ) =
M ′ .
The rank of the matrix B in turn determines the complexity of connectivity matrices D EE and D IEthrough eq.
In the simplest case (beyond the trivial case where all elements of B are set to zero), Γ is givenby a single column vector, Γ = b 1 , which sets the matrix B to the following: B = a b 1 b 1 ⊺ , andthe rank of the matrix B to rank ( B ) = 1 . We simulate such network, setting the membrane timeconstants of E and I cell type to biologically plausible values, with λ − 1 E = 10 ms for pyramidalneurons, and λ − 1 I = 5 ms for inhibitory interneurons, as found in biological data [42]. In addition,we assumed the time constants of the population read-out to be shorter (specifically, twice as short)than the time constant of the single neuron read-out, as single-neuron read-out is a slower processof homeostatic regulation of the firing rate of single neurons (see eq. 4). According to eq.
(16), thisgives spike-triggered adaptation in E and I neurons. We found a population-wide oscillation anda network response that outlast the external stimulation (Figure 1C), in contrast to the response ofthe simpler network ((Figure 1B)).
Similarly to the simpler network, we still have that only neuronsaligned with the input feature activate (e.g., neurons with w Y 1 i > 0 with s 1 ( t ) > 0 ). By increasing the rank of the matrix B , also neurons that are not aligned with the input featureparticipate in the network response. We now define the matrix B with M ′ > 1 linearly independentvectors, where a 1 b 1 + · · · + a M ′ b M ′
=
if and only if a n = 0 ∀ n . Linear independence of columnvectors b n implies that the matrix B has rank M ′ > 1 . We find that with M ′ > 1 , both neurons with A
B
C s(t) x(t)
w ET x ̂ E (t) x
̂ I (t)
C EI
C IE
C II w E w I
D EE
D IE = 0 . 25 ∀ i . Other parameters as in B. decoding vector aligned as well as misaligned with the input feature respond (Figure 2A-B; withs 1 ( t ) > 0 , neurons with w Y 1 i > 0 as well as with w Y 1 i > 0 are active).
Duration of network’s responsedepends on the strength of E-E synaptic connectivity, here controlled by the parameter a in eq.
With sufficiently strong parameter a , the network maintains sustained activity (Figure 2B).
In theabsence of external drive, but with sufficient noise at threshold that by itself occasionally leads to aspike (see eq. 10), such network responds with Up and Down states that are triggered by the noiseand amplified by the recurrent excitatory synaptic activity (Figure 2C).
[Figure]
Discussion
We developed an E-I spiking network that optimizes information coding.
The optimization is basedon minimizing two cost functions comprising a quadratic estimation error and a quadratic regularizerand assuming an arbitrary linear input/output transformation.
The general solution of an on-lineminimization of cost functions with spikes is a generalized LIF model with a specific set of structuraland dynamical features.
Structural features include low-rank structured connectivity [40, 18, 20],where fast synapses between E and I neurons implement competition between neurons with similardecoding selectivity and slower synapses implement cooperation.
Moreover, structural featuresresulting from optimization with spikes are external inputs that are shared across neurons, spiketriggered adaptation currents in single neurons and hyperpolarization-triggered rebound current in Eneurons.
Assuming heterogeneous decoding weights, all currents with the exception of leak currentsare heterogeneous across neurons, increasing the biological plausibility of the model.
Similarly to the seminal paper by Boerlin and colleagues [2], the spiking network proposed hereassumes that spikes are fired only if they minimize the quadratic error between the signal and thepopulation read-out.
As the network model is derived from such s cost function, efficient codingis a build-in property of the network. In earlier works on efficient coding [4], however, the leakcurrent is not derived exactly but only added to the membrane equation for biological plausibility. Our derivations define an exact expression for the leak current, similarly as in [19]. Moreover, inmost previous works on efficient coding with spikes, a single cost function is used to derive themembrane equations.
Here, we propose two separate cost functions that relate to the activity of Eand I neurons.
A previous proposal that already used separate cost functions for E and I neurons [6]yielded a network similar to our simplified version of the network, but the behavior or their network ismarkedly different than our simplified model, due to very long and unrealistic synaptic time constantsthey use. Most previous proposals on efficient coding [5, 6, 19, 4] assume a specific computation, namelya leaky integration of external inputs (but see [2]). We here developed the case of the generallinear transformation between the external inputs and the internal signals of the network with abiologically plausible spiking network. This generalization allowed us to formulate a biologicallyplausible expression for E-E connections and to gain insights on how the structure of the mixingmatrix qualitatively changes network responses.
Without mixing of features, only neurons that arealigned to the external stimulus, and consequently driven by the feedforward current, participate innetwork responses.
With increased complexity of E-E connections, we also observe activation ofneurons that are not driven by the feedforward current, but are instead engaged through recurrent E-Econnections.
Moreover, in the absence of the external stimulus and with sufficient background noise,such a complex network produces spontaneous Up and Down states that could describe spontaneous Up and Down states observed in biological networks [39].
As it follows from our results, the minimization of the quadratic cost function with spikes resultsin a generalized LIF model, a spiking neuron model that has been shown to provide a good fitto the spiking dynamics of biological networks [12, 14].
An important advantage of the presentcomputational framework over previously proposed generalized LIF models is the access to afunctional interpretation of network parameters.
In general, a change in a network parametercan have a strong influence on network’s dynamics.
Besides characterizing the effect on neuraldynamics, it is insightful to also have the interpretation of the role of a specific parameter on thefunctional/computational level.
In the present setting, parameters of the spiking network are directlyrelated to the variables minimized in cost functions, which allows to connect their effect on dynamicswith their role for network’s computation.
In the present work, we emphasized the theoretical development of an optimization-based approachtowards a biologically plausible spiking neural network with complex responses.
Future work couldintroduce performance measures, for example by measuring the distance between the signal andthe population read-out, or by using information-theoretic measures [3, 27, 35].
Moreover, futurework could perform an analysis of network parameters and assess how are particular dynamicalphenomena, such as spontaneous Up and Down states, related to the information processing in thespiking network.
Finally, the framework can possibly be extended to include further biophysicalproperties of cortical networks, such as spatial extension of tuning curves, several types of inhibitoryneurons [45], or non-linear computations [41].