Original: L EARNING

Original: D IFFICULT

Original:  P ERCEPTUAL T ASKS WITH H ODGKIN -H UXLEY N ETWORKS
Revised: L EARNING H IGH -D IMENSIONAL P ERCEPTUAL T ASKS WITH H ODGKIN -H UXLEY N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper demonstrates that a computational neural network model using ion channel-based conductances to transmit information can solve standard computer vision datasets at near state-of-the-art performance.
Revised: This paper demonstrates that a computational neural network model using ion channel-based conductances to transmit information can solve standard computer vision datasets at near state-of-the-art performance.

Original: Although not fully biologically accurate, this model incorporates fundamental biophysical principles underlying the control of membrane potential and the processing of information by Ohmic ion channels.
Revised: Although not fully biologically accurate, this model incorporates fundamental biophysical principles underlying the control of membrane potential and the processing of information by Ohmic ion channels.

Original: The key computational step employs Conductance-Weighted Averaging (CWA) in place of the traditional afﬁne transformation, representing a fundamentally different computational principle.
Revised: The key computational step employs Conductance-Weighted Averaging (CWA) in place of the traditional afﬁne transformation, representing a fundamentally different computational principle.

Original: Importantly, CWA based networks are self-normalizing and range-limited.
Revised: Importantly, CWA based networks are self-normalizing and range-limited.

Original: We also demonstrate for the ﬁrst time that a network with excitatory and inhibitory neurons and nonnegative synapse strengths can successfully solve computer vision problems.
Revised: We also demonstrate for the ﬁrst time that a network with excitatory and inhibitory neurons and nonnegative synapse strengths can successfully solve computer vision problems.

Original: Although CWA models do not yet surpass the current state-of-the-art in deep learning, the results are competitive on CIFAR-10.
Revised: Although CWA models do not yet surpass the current state-of-the-art in deep learning, the results are competitive on CIFAR-10.

Original: There remain avenues for improving these networks, e g. , by more closely modeling ion channel function and connectivity patterns found in the brain.
Revised: There remain avenues for improving these networks, e.g. , by more closely modeling ion channel function and connectivity patterns found in the brain.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has achieved groundbreaking advances in computer vision, but modern artiﬁcial neural networks have largely developed independent of neuroscience research on biological neuronal networks.
Revised: Deep learning has achieved groundbreaking advances in computer vision, but modern artiﬁcial neural networks have largely developed independent of neuroscience research on biological neuronal networks.

Original: Many of the core features found in biological neural networks have never been integrated successfully into artiﬁcial neural networks, raising serious questions of whether the success of deep learning is relevant to neuroscience research or vice versa .
Revised: Many of the core features found in biological neural networks have never been integrated successfully into artiﬁcial neural networks, raising serious questions of whether the success of deep learning is relevant to neuroscience research or vice versa .

Original: This paper presents an artiﬁcial neuronal network incorporating features found in the classic HodgkinHuxley equations that relate neuronal membrane potential to the activity of ion channel conductance pathways.
Revised: This paper presents an artiﬁcial neuronal network incorporating features found in the classic HodgkinHuxley equations that relate neuronal membrane potential to the activity of ion channel conductance pathways.

Original: A primary feature of this approach is that the values that the neuronal elements of the network can take are range-limited and self-normalizing.
Revised: A primary feature of this approach is that the values that the neuronal elements of the network can take are range-limited and self-normalizing.

Original: In addition, each neuron is classiﬁed as either excitatory or inhibitory, an aspect of biological neural networks that until now has never been effectively utilized in artiﬁcial neural networks.
Revised: In addition, each neuron is classiﬁed as either excitatory or inhibitory, an aspect of biological neural networks that until now has never been effectively utilized in artiﬁcial neural networks.

Original: Here we demonstrate that more biologically accurate networks can learn complex computer vision datasets such as CIFAR-10 competitively with the state-of-the-art just a few years prior.
Revised: Here we demonstrate that more biologically accurate networks can learn complex computer vision datasets such as CIFAR-10 competitively with the state-of-the-art just a few years prior.

Original: Biological neurons do not compute an afﬁne transform of their inputs.
Revised: Biological neurons do not compute an afﬁne transform of their inputs.

Original: Rather, the synaptic inputs compete against other active conductance pathways in the neuron for the control of the membrane potential and thus the ﬁring state of the neuron.
Revised: Rather, the synaptic inputs compete against other active conductance pathways in the neuron for the control of the membrane potential and thus the ﬁring state of the neuron.

Original: The valence of these different pathways is reﬂected by the zero-current potentials of the types of channel conductances being regulated, which determines whether the pathway excites or inhibits the neuron.
Revised: The valence of these different pathways is reﬂected by the zero-current potentials of the types of channel conductances being regulated, which determines whether the pathway excites or inhibits the neuron.

Original: The combination of these conductances into a single membrane potential is analogous to performing a conductance weighted averaging (CWA) of these zero-current potentials.
Revised: The combination of these conductances into a single membrane potential is analogous to performing a conductance weighted averaging (CWA) of these zero-current potentials.

Original: The concept of neuron valence is related to the types of synaptic channels that a neuron regulates in its post-synaptic neurons.
Revised: The concept of neuron valence is related to the types of synaptic channels that a neuron regulates in its post-synaptic neurons.

Original: Because neurons primarily release one type of neurotransmitter, they functionally fall into one of two classes: Excitatory or Inhibitory neurons.
Revised: Because neurons primarily release one type of neurotransmitter, they functionally fall into one of two classes: Excitatory or Inhibitory neurons.

Original: In this paper, CWA similarly uses two valences, (+1) for neurons forming excitatory connections and (-1) for neurons forming inhibitory connections.
Revised: In this paper, CWA similarly uses two valences, (+1) for neurons forming excitatory connections and (-1) for neurons forming inhibitory connections.

Original: Importantly, the combining of these pathways into a single neuron using CWA is constrained to produce neuronal outputs that lie somewhere between these two extremes, and thus the output of CWA is bound by the range of the valences.
Revised: Importantly, the combining of these pathways into a single neuron using CWA is constrained to produce neuronal outputs that lie somewhere between these two extremes, and thus the output of CWA is bound by the range of the valences.

Original: Normalization and regularization strategies, such as BatchNorm (Ioffe & Szegedy, 2015) or
Revised: Normalization and regularization strategies, such as BatchNorm (Ioffe & Szegedy, 2015) or

Original: DropOut (Srivastava et al., 2014), have been repeatedly shown to improve the quality of artiﬁcial neural networks.
Revised: DropOut (Srivastava et al., 2014), have been repeatedly shown to improve the quality of artiﬁcial neural networks.

Original: But CWA networks are by deﬁnition normalized and range-limited.
Revised: But CWA networks are by deﬁnition normalized and range-limited.

Original: Therefore, one can conjecture that CWA plays a normalization role in biological neural networks.
Revised: Therefore, one can conjecture that CWA plays a normalization role in biological neural networks.

Original: As seen in
Revised: As seen in

Original: Table 4.3, bare CWA without additional normalization or regularization is competitive with standard afﬁne networks that employ these strategies, especially in the case of fully connected networks (FCN), whose neurons, like biological neurons, are characterized by having thousands of inputs.
Revised: Table ?? , bare CWA without additional normalization or regularization is competitive with standard afﬁne networks that employ these strategies, especially in the case of fully connected networks (FCN), whose neurons, like biological neurons, are characterized by having thousands of inputs.

Original: In sum, neural networks with CWA provide a more biologically accurate model with excitatoryinhibitory neurons and conductance-based normalization that succeeds on computer vision datasets without extra normalization and regularization, with implications for neuroscience and deep learning.
Revised: In sum, neural networks with CWA provide a more biologically accurate model with excitatoryinhibitory neurons and conductance-based normalization that succeeds on computer vision datasets without extra normalization and regularization, with implications for neuroscience and deep learning.

Original:  A BSTRACTING F ROM H ODGKIN -H UXLEY
Revised:  A BSTRACTING F ROM H ODGKIN -H UXLEY

Original: Hodgkin and Huxley described the change in voltage within a neuron as a function of the ion channels active in the neuron (Hodgkin & Huxley, 1952), for which they received a Nobel prize.
Revised: Hodgkin and Huxley described the change in voltage within a neuron as a function of the ion channels active in the neuron (Hodgkin & Huxley, 1952), for which they received a Nobel prize.

Original: Their model remains the foundation of a modern understanding of biological neurons, but deep learning was never based directly on this model.
Revised: Their model remains the foundation of a modern understanding of biological neurons, but deep learning was never based directly on this model.

Original: In this section, a steady-state abstraction of the Hodgkin-Huxley model is developed, compared to standard ( i.e. , afﬁne) artiﬁcial neural networks, and used to derive the
Revised: In this section, a steady-state abstraction of the Hodgkin-Huxley model is developed, compared to standard ( i.e. , afﬁne) artiﬁcial neural networks, and used to derive the

Original: CWA neuron as an abstraction.
Revised: CWA neuron as an abstraction.

Original: 2. S TEADY -S TATE C ONDUCTANCE M ODEL
Revised: 2. S TEADY -S TATE C ONDUCTANCE M ODEL

Original: The information state of a neuron is encoded by the voltage on its surface membrane.
Revised: The information state of a neuron is encoded by the voltage on its surface membrane.

Original: The surface membrane is generally analogous to a capacitor; on its surface are many ion channels that transfer charge across this capacitance, allowing the movement of ions into or out of the cell.
Revised: The surface membrane is generally analogous to a capacitor; on its surface are many ion channels that transfer charge across this capacitance, allowing the movement of ions into or out of the cell.

Original: Depending on the ion species of a channel, the membrane potential will move in a direction determined by the equilibrium potential of that species.
Revised: Depending on the ion species of a channel, the membrane potential will move in a direction determined by the equilibrium potential of that species.

Original: Ion ﬂow changes the voltage across the membrane and is the key mechanism of information transfer within the brain.
Revised: Ion ﬂow changes the voltage across the membrane and is the key mechanism of information transfer within the brain.

Original: The strength of a synapse depends on the number of ion channels at the synapse, the ion species of the channel, and the overall activity of other pathways.
Revised: The strength of a synapse depends on the number of ion channels at the synapse, the ion species of the channel, and the overall activity of other pathways.

Original: Electrochemically, the net effect is a conductance-weighted average of the equilibrium potentials for all the ion channel pathways in the neuron (see e.g. Kandel et al.
Revised: Electrochemically, the net effect is a conductance-weighted average of the equilibrium potentials for all the ion channel pathways in the neuron (see e.g. Kandel et al.

Original: (2013), Ch. 5).
Revised: (2013), Ch. 5).

Original: In the Hodgkin-Huxley model, the membrane potential (the voltage difference at the membrane) always moves in the direction that will reduce the net ionic current towards zero.
Revised: In the Hodgkin-Huxley model, the membrane potential (the voltage difference at the membrane) always moves in the direction that will reduce the net ionic current towards zero.

Original: The voltage and current are time-varying quantities governed by a differential equation, but they can be converted to a time-invariant, steady-state model by presuming that the the inputs to the network are constant.
Revised: The voltage and current are time-varying quantities governed by a differential equation, but they can be converted to a time-invariant, steady-state model by presuming that the the inputs to the network are constant.

Original: Because ion movement is linked to changes in the charge of the membrane capacitance, the steadystate membrane potential is found by letting the net ionic current go to zero.
Revised: Because ion movement is linked to changes in the charge of the membrane capacitance, the steadystate membrane potential is found by letting the net ionic current go to zero.

Original: For linear Ohmic conductance pathways, this stable membrane potential is determined by the conductance-weighted average of the zero-current potentials of the ionic pathways present in the neuron.
Revised: For linear Ohmic conductance pathways, this stable membrane potential is determined by the conductance-weighted average of the zero-current potentials of the ionic pathways present in the neuron.

Original: For a single-chamber neuron i , and letting j ∼ i range over presynaptic neurons j , the voltage V i of i can be written in simpliﬁed form as:
Revised: For a single-chamber neuron i , and letting j ∼ i range over presynaptic neurons j , the voltage V i of i can be written in simpliﬁed form as:

Original: [Equation]
Revised: [Equation]

Original: where pj =
Revised: p

Original: [Equation]
Revised: [Equation]

Original: This steady-state equation is the expected value of a probabilistic model driven by neural spike trains.
Revised: This steady-state equation is the expected value of a probabilistic model driven by neural spike trains.

Original: The value p j represents the probability that the pathways leading out of neuron j are open, that is, the synapses out of j are releasing transmitter onto the post-synaptic ion channels.
Revised: The value p j represents the probability that the pathways leading out of neuron j are open, that is, the synapses out of j are releasing transmitter onto the post-synaptic ion channels.

Original: The exact computation of p j is the only element of Equation 1 not well deﬁned by biophysics; in steady-state (non-spiking) models, it is commonly modeled as a logistic function or a ReLU.
Revised: The exact computation of p j is the only element of Equation 1 not well deﬁned by biophysics; in steady-state (non-spiking) models, it is commonly modeled as a logistic function or a ReLU.

Original: The logistic function in Equationhas a spike threshold E th j and is scaled by a ﬁxed scaling constant k j .
Revised: The logistic function in Equationhas a spike threshold E th j and is scaled by a ﬁxed scaling constant k j .

Original: The G i,j terms describe the maximum conductance possible along the pathway from neuron j to i , which occurs when p j = 1 .
Revised: The G i,j terms describe the maximum conductance possible along the pathway from neuron j to i , which occurs when p j = 1 .

Original: The term G r represents the resting self-conductance of the neuron, and E r the self-potential absent synaptic activity (see e.g. Gabbiani & Cox (2017), Chs. 2,11;
Revised: The term G r represents the resting self-conductance of the neuron, and E r the self-potential absent synaptic activity (see e.g. Gabbiani & Cox (2017), Chs. 2,11;

Original: Koch & Segev (1999), Chs. 1,2).
Revised: Koch & Segev (1999), Chs. 1,2).

Revised: These interactions are shown in Figure 1.

Original: The E j terms are the zero-current potentials for each pathway, speciﬁc to the type of neuron j .
Revised: The E j terms are the zero-current potentials for each pathway, speciﬁc to the type of neuron j .

Original: For example, pyramidal neurons produce excitatory glutamatergic synapses yielding larger E j values, but lateral inhibitory interneurons produce GABA-ergic synapses with E
Revised: For example, pyramidal neurons produce excitatory glutamatergic synapses yielding larger E j values, but lateral inhibitory interneurons produce GABA-ergic synapses with E

Original: j values below the threshold.
Revised: j values below the threshold.

Original: At behaviorally relevant time scales, changes in p j are solely responsible for neuronal membrane potential dynamics.
Revised: At behaviorally relevant time scales, changes in p j are solely responsible for neuronal membrane potential dynamics.

Original: Key factors causing changes in p j include changes in synaptic activity for neurotransmitter-gated channels or changes in gating of the other membrane channels due to voltage or intracellular messengers.
Revised: Key factors causing changes in p j include changes in synaptic activity for neurotransmitter-gated channels or changes in gating of the other membrane channels due to voltage or intracellular messengers.

Original: For neurons, the importance of these changes in membrane potential are that they regulate neuronal spiking and hence neurotransmitter release onto follower neurons.
Revised: For neurons, the importance of these changes in membrane potential are that they regulate neuronal spiking and hence neurotransmitter release onto follower neurons.

Original: At larger time scales, the G i,j adapt to control how the neuron processes information based on the addition or removal of channels; the method of this adaptation is as yet not well understood.
Revised: At larger time scales, the G i,j adapt to control how the neuron processes information based on the addition or removal of channels; the method of this adaptation is as yet not well understood.

Original: All of the terms in Equation 1 have ﬁxed sign: G i,j , p j , E j , G r , E r , E th j , and k j .
Revised: All of the terms in Equation 1 have ﬁxed sign: G i,j , p j , E j , G r , E r , E th j , and k j .

Original: In this model, a synapse is inhibitory if the incoming zero-current potential E j is less than the outgoing threshold
Revised: In this model, a synapse is inhibitory if the incoming zero-current potential E j is less than the outgoing threshold

Original: E th i , in which case j has an inﬂuence on i that tends to push the spike probability below 12 .
Revised: E th i , in which case j has an inﬂuence on i that tends to push the spike probability below 12 .

Original: Thus neuron types with E j less than the threshold are described as inhibitory , and those with E j above the threshold are excitatory .
Revised: Thus neuron types with E j less than the threshold are described as inhibitory , and those with E j above the threshold are excitatory .

Original: Biological neural networks have several types of inhibitory and excitatory neurons, but every outgoing synapse of an excitatory neuron has an excitatory effect on the postsynaptic neuron, and conversely every outgoing synapse of an inhibitory neuron has an inhibitory effect.
Revised: Biological neural networks have several types of inhibitory and excitatory neurons, but every outgoing synapse of an excitatory neuron has an excitatory effect on the postsynaptic neuron, and conversely every outgoing synapse of an inhibitory neuron has an inhibitory effect.

Original: This partitions the network into classes of neurons with valences E j determined by the class.
Revised: This partitions the network into classes of neurons with valences E j determined by the class.

Original: 2. C OMPARISON TO A FFINE N EURAL N ETWORKS
Revised: 2. C OMPARISON TO A FFINE N EURAL N ETWORKS

Original: Artiﬁcial neural networks were originally modeled on biological neurons (Rosenblatt, 1958) but with simpliﬁcations.
Revised: Artiﬁcial neural networks were originally modeled on biological neurons (Rosenblatt, 1958) but with simpliﬁcations.

Original: In particular, they assume that the total conductance is ﬁxed (Shriki et al., 2003).
Revised: In particular, they assume that the total conductance is ﬁxed (Shriki et al., 2003).

Original: In analogy to Equation 1, the typical artiﬁcial neuron is described by
Revised: In analogy to Equation 1, the typical artiﬁcial neuron is described by

Original: [Equation]
Revised: [Equation]

Original: where σ is an activation function, typically a sigmoid such as the logistic or tanh .
Revised: where σ is an activation function, typically a sigmoid such as the logistic or tanh .

Original: Because z i is an afﬁne transformation of the inputs p j , such networks will be referred to as afﬁne networks below.
Revised: Because z i is an afﬁne transformation of the inputs p j , such networks will be referred to as afﬁne networks below.

Original: Due to the long history of afﬁne networks within machine learning, their connection with neuroscience is now obsolete, with advances in neuroscience having relatively impact on the machine learning community.
Revised: Due to the long history of afﬁne networks within machine learning, their connection with neuroscience is now obsolete, with advances in neuroscience having relatively impact on the machine learning community.

Original: The disconnection is further enhanced by the success of deep learning in computer vision and natural language processing, in comparison to neuroscience simulations which typically just seek to mimic some aspect of experimental observations.
Revised: The disconnection is further enhanced by the success of deep learning in computer vision and natural language processing, in comparison to neuroscience simulations which typically just seek to mimic some aspect of experimental observations.

Original: One of the goals of this paper is to demonstrate experimentally that networks of CWA neurons are potentially as powerful as afﬁne neural networks.
Revised: One of the goals of this paper is to demonstrate experimentally that networks of CWA neurons are potentially as powerful as afﬁne neural networks.

Original: Equation 2 has similarities to Equation 1, but two aspects of Hodgkin-Huxley model are missing:
Revised: Equation 2 has similarities to Equation 1, but two aspects of Hodgkin-Huxley model are missing:

Original: A conductance-weighted average (CWA) controlling neuronal activity.
Revised: A conductance-weighted average (CWA) controlling neuronal activity.

Original: A zero-current potential making neurons either excitatory or inhibitory.
Revised: A zero-current potential making neurons either excitatory or inhibitory.

Original: With regard to (1), conductance-weighted averaging has a normalizing inﬂuence, limiting voltage to a range determined by the zero-current potentials E j .
Revised: With regard to (1), conductance-weighted averaging has a normalizing inﬂuence, limiting voltage to a range determined by the zero-current potentials E j .

Original: Numerous methods in deep learning impose normalization on neural activation, but this biological mechanism has not been extensively explored.
Revised: Numerous methods in deep learning impose normalization on neural activation, but this biological mechanism has not been extensively explored.

Original: The nature of the CWA operation is fundamentally distinct from the operation of an afﬁne net.
Revised: The nature of the CWA operation is fundamentally distinct from the operation of an afﬁne net.

Original: In
Revised: In

Original: Equation 1, the output p j of pre-synaptic neurons is only used in order to determine the conductance weighting, and the post-synaptic voltage is the weighted average of the incoming zero-current potentials E j , which divides the neurons into excitatory and inhibitory classes, as in point (2) above.
Revised: Equation 1, the output p j of pre-synaptic neurons is only used in order to determine the conductance weighting, and the post-synaptic voltage is the weighted average of the incoming zero-current potentials E j , which divides the neurons into excitatory and inhibitory classes, as in point (2) above.

Original: Previous attempts to implement Equation 1 have been limited in scope and results.
Revised: Previous attempts to implement Equation 1 have been limited in scope and results.

Original: We demonstrate that an abstraction incorporating points (1) and (2) above can solve computer vision problems to a level that exceeds the state-of-the-art in afﬁne networks from just a few years prior.
Revised: We demonstrate that an abstraction incorporating points (1) and (2) above can solve computer vision problems to a level that exceeds the state-of-the-art in afﬁne networks from just a few years prior.

Original: 2. C ONDUCTANCE -W EIGHTED A VERAGING
Revised: 2. C ONDUCTANCE -W EIGHTED A VERAGING

Original: From Equation 1, the Hodgkin-Huxley neuron computes a weighted sum of the ﬁxed E j values, i.e. ,
Revised: From Equation 1, the Hodgkin-Huxley neuron computes a weighted sum of the ﬁxed E j values, i.e. ,

Original: V i = (cid:88)
Revised: V i = (cid:88)

Original: j ∼ i w i,j E j where w i,j =
Revised: j ∼ i w i,j E j where w i,j =

Original: G i,j p j (cid:80) k ∼ i G i,k p k
Revised: G i,j p j (cid:80) k ∼ i G i,k p k

Original: Here w i,j reﬂects both inputs and parameters .
Revised: Here w i,j reﬂects both inputs and parameters .

Original: For an input vector p this equation computes a probability distribution µ ( p ) and takes the expected value over a vector of ﬁxed values, labeled e : 
Revised: For an input vector p this equation computes a probability distribution µ ( p ) and takes the expected value over a vector of ﬁxed values, labeled e : z i = E µ ( p ) [ e ] = (cid:88) j ∼ i e [Equation]

Revised: p )

Original: [Equation]
Revised: [Equation]

Original:  The parameters g ij ∈ R vary across the real line, and the function h : R → [0 , ∞ )
Revised: (cid:88) [Equation] The parameters g ij ∈ R vary across the real line, and the function h : R → [0 , ∞ )

Original: enforces nonnegativity.
Revised: enforces nonnegativity.

Original: Equation 1 is obtained by setting G i,j = h ( g ij ) and supposing a connection to a bias node r with h ( g ir )
Revised: Equation 1 is obtained by setting G i,j = h ( g ij ) and supposing a connection to a bias node r with h ( g ir )

Original: =
Revised: =

Original: G r with p r = 1 .
Revised: G r with p r = 1 .

Original: Given that p j ∈ [0 , 1] , it follows that z i is necessarily range-limited, falling between the minimum and maximum values of e j .
Revised: Given that p j ∈ [0 , 1] , it follows that z i is necessarily range-limited, falling between the minimum and maximum values of e j .

Original: For simplicity, set
Revised: For simplicity, set

Original: [Equation]
Revised: [Equation]

Original: so that e j and z i fall in the interval [ − 1 , 1] .
Revised: so that e j and z i fall in the interval [ − 1 , 1] .

Original: Equation 3 is
Revised: Equation 3 is

Original: Conductance-Weighted Averaging or CWA.
Revised: Conductance-Weighted Averaging or CWA.

Original: Notice that Equation 3 is self-normalizing.
Revised: Notice that Equation 3 is self-normalizing.

Original: The output z i necessarily lies in the range spanned by the e j , so if e j ∈ [ − 1 , 1] , then z i ∈ [ − 1 , 1] .
Revised: The output z i necessarily lies in the range spanned by the e j , so if e j ∈ [ − 1 , 1] , then z i ∈ [ − 1 , 1] .

Original: The factor µ ( p ) j imposes normalization.
Revised: The factor µ ( p ) j imposes normalization.

Original: The dependence on p j determines which of the e j to listen to, and the output is a weighted average of these values.
Revised: The dependence on p j determines which of the e j to listen to, and the output is a weighted average of these values.

Original: Afﬁne neural networks alternate between afﬁne transformation and non-linear activation.
Revised: Afﬁne neural networks alternate between afﬁne transformation and non-linear activation.

Original: Equationis already a non-linear transformation due to the denominator, and one might wonder whether non-linear activation is then necessary.
Revised: Equationis already a non-linear transformation due to the denominator, and one might wonder whether non-linear activation is then necessary.

Original: In preliminary experiments, omitting a further activation function was sufﬁcient to solve the MNIST digits problem with 98% accuracy with a fully connected architecture, but obtaining good performance with more complex architectures and datasets did in fact require non-linear activation to silence minimally activated neurons, providing the beneﬁt of sparse neural activation, known to contribute to good generalization in afﬁne networks.
Revised: In preliminary experiments, omitting a further activation function was sufﬁcient to solve the MNIST digits problem with 98% accuracy with a fully connected architecture, but obtaining good performance with more complex architectures and datasets did in fact require non-linear activation to silence minimally activated neurons, providing the beneﬁt of sparse neural activation, known to contribute to good generalization in afﬁne networks.

Original: Rather than implementing p i as a sigmoid based on E th i and k j , we observe that the biophysics is not speciﬁc here and employ a simpler scaling and clipping function:
Revised: Rather than implementing p i as a sigmoid based on E th i and k j , we observe that the biophysics is not speciﬁc here and employ a simpler scaling and clipping function:

Original: [Equation]
Revised: [Equation]

Original: This equation is justiﬁed by the fact that as an expected value z i is bounded by the ﬁxed values of the vector e so that z i ∈ [ − 1 , 1] .
Revised: This equation is justiﬁed by the fact that as an expected value z i is bounded by the ﬁxed values of the vector e so that z i ∈ [ − 1 , 1] .

Original: Clipping off the negative values yields p i ∈ [0 , 1] as desired; it also results in ReLU activation (Nair & Hinton, 2010).
Revised: Clipping off the negative values yields p i ∈ [0 , 1] as desired; it also results in ReLU activation (Nair & Hinton, 2010).

Original: The biological appropriateness of this simpliﬁcation would depend on (1) the extent to which the spiking probability of biological neurons approximates a clipped linear function of membrane potential, and (2) whether there is functional signiﬁcance to the balance of positive and negative e j with respect to the spike threshold in biological systems.
Revised: The biological appropriateness of this simpliﬁcation would depend on (1) the extent to which the spiking probability of biological neurons approximates a clipped linear function of membrane potential, and (2) whether there is functional signiﬁcance to the balance of positive and negative e j with respect to the spike threshold in biological systems.

Original: These questions will not be resolved here, but a different biological activation function would likely not substantially change the overall computations resulting from Equations 3 and 4.
Revised: These questions will not be resolved here, but a different biological activation function would likely not substantially change the overall computations resulting from Equations 3 and 4.

Original: Now suppose in the extreme that rather than choosing e j anywhere in the interval [ − 1 , 1] , one enforces either e j
Revised: Now suppose in the extreme that rather than choosing e j anywhere in the interval [ − 1 , 1] , one enforces either e j

Original: = +1 or e j = − 1 .
Revised: = +1 or e j = − 1 .

Original: If e j = +1 neuron j is excitatory or positive , and if e j = − 1 , neuron j is inhibitory or negative .
Revised: If e j = +1 neuron j is excitatory or positive , and if e j = − 1 , neuron j is inhibitory or negative .

Original: This restriction loses no mathematical power of the system.
Revised: This restriction loses no mathematical power of the system.

Original: As long as µ ( p ) can represent (nearly) arbitrary probability distributions, then z i can result in any point in the interval [ − 1 , 1] using just e j ∈ {− 1 , +1 } .
Revised: As long as µ ( p ) can represent (nearly) arbitrary probability distributions, then z i can result in any point in the interval [ − 1 , 1] using just e j ∈ {− 1 , +1 } .

Original: Of course, in biological neural networks, there are several distinct values for e j , not just two.
Revised: Of course, in biological neural networks, there are several distinct values for e j , not just two.

Original: It is not clear yet whether this distinction is computationally signiﬁcant or merely a result of biological accident.
Revised: It is not clear yet whether this distinction is computationally signiﬁcant or merely a result of biological accident.

Original: Nonetheless, in implementing CWA, the experiments below divide the neurons up into two groups of positive/excitatory and negative/inhibitory neurons.
Revised: Nonetheless, in implementing CWA, the experiments below divide the neurons up into two groups of positive/excitatory and negative/inhibitory neurons.

Original: To explain these networks, Equation 3 can be rewritten as
Revised: To explain these networks, Equation 3 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where N i ( p ) = (cid:80) j p j h ( g ij ) is the normalizing value.
Revised: where N i ( p ) = (cid:80) j p j h ( g ij ) is the normalizing value.

Original: From an intuitive perspective, the left term as sums up the positive evidence and the right term sums the negative evidence.
Revised: From an intuitive perspective, the left term as sums up the positive evidence and the right term sums the negative evidence.

Original: If the negative evidence predominates, then z i is negative.
Revised: If the negative evidence predominates, then z i is negative.

Original: If the positive evidence predominates, then z i is positive.
Revised: If the positive evidence predominates, then z i is positive.

Original: The parameters g ij control which evidence is listened to by neuron i .
Revised: The parameters g ij control which evidence is listened to by neuron i .

Original: Once z i is thought of as computing evidence for something , the role of neuron i within a larger network is to represent that something as a discrete feature.
Revised: Once z i is thought of as computing evidence for something , the role of neuron i within a larger network is to represent that something as a discrete feature.

Original: If z i > 0 , the feature is observed to a degree determined by the magnitude of z i .
Revised: If z i > 0 , the feature is observed to a degree determined by the magnitude of z i .

Original: If z i ≤ 0 , then on balance the feature has not been observed.
Revised: If z i ≤ 0 , then on balance the feature has not been observed.

Original: To summarize, CWA is an abstraction based on the Hodgkin-Huxley model.
Revised: To summarize, CWA is an abstraction based on the Hodgkin-Huxley model.

Original: Its purpose is to demonstrate that networks using conductance-weighted averaging and partitioned into excitatory and inhibitory neurons can solve complex computer vision datasets.
Revised: Its purpose is to demonstrate that networks using conductance-weighted averaging and partitioned into excitatory and inhibitory neurons can solve complex computer vision datasets.

Original: This is a more realistic model of biological neural networks than the afﬁne model, but abstracted from reality in the following ways:
Revised: This is a more realistic model of biological neural networks than the afﬁne model, but abstracted from reality in the following ways:

Original: 1) Non-synaptic channels ( i.e. , G r ) are not dynamically gating and so their p values are unversally 1 .
Revised: 1) Non-synaptic channels ( i e. , G r ) are not dynamically gating and so their p values are unversally 1 .

Original: They behave similarly to the bias term in afﬁne networks but also provide a mathematical anchor point for the normalizing sum.
Revised: They behave similarly to the bias term in afﬁne networks but also provide a mathematical anchor point for the normalizing sum.

Original: 2) Values for the zero-current potentials are scaled to lie in the range of ± 1 , and the channel conductances G i,j are represented by real-valued proxies g ij .
Revised: 2) Values for the zero-current potentials are scaled to lie in the range of ± 1 , and the channel conductances G i,j are represented by real-valued proxies g ij .

Original: Synaptic channel activity probability for a neuron is controlled by the membrane potential of the presynaptic neuron, and is modeled by a typical computational neuronal network activity function as in Equation 4 acting on the presynaptic neuron’s membrane potential.
Revised: Synaptic channel activity probability for a neuron is controlled by the membrane potential of the presynaptic neuron, and is modeled by a typical computational neuronal network activity function as in Equation 4 acting on the presynaptic neuron’s membrane potential.

Original: Only two types of zero-current potentials are used, +1 for excitatory neurons and − 1 for inhibitory neurons.
Revised: Only two types of zero-current potentials are used, +1 for excitatory neurons and − 1 for inhibitory neurons.

Original: The next sections discuss related work and the provide experimental results for CWA on computer vision tasks using classic fully-connected and convolutional architectures.
Revised: The next sections discuss related work and the provide experimental results for CWA on computer vision tasks using classic fully-connected and convolutional architectures.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Spiking Neural Networks.
Revised: Spiking Neural Networks.

Original: Spiking neural network (SNN) models simulating the membrane potential as a differential equation have been applied to classiﬁcation problems Pfeiffer & Pfeil (2018).
Revised: Spiking neural network (SNN) models simulating the membrane potential as a differential equation have been applied to classiﬁcation problems Pfeiffer & Pfeil (2018).

Original: The best supervised test accuracy for SNNs on MNIST is 99.31% with a convolutional SNN trained using backpropagation Lee et al.
Revised: The best supervised test accuracy for SNNs on MNIST is 99.31% with a convolutional SNN trained using backpropagation Lee et al.

Original: Several methods for SNNs achieve 98-99% on MNIST (Kulkarni & Rajendran, 2018; Diehl et al., 2015; Kheradpisheh et al., 2018).
Revised: Several methods for SNNs achieve 98-99% on MNIST (Kulkarni & Rajendran, 2018; Diehl et al., 2015; Kheradpisheh et al., 2018).

Original: SNNs have also been run on
Revised: SNNs have also been run on

Original: CIFAR-10 with test accuracy of up to 92.37% (Hu et al., 2018; Sengupta et al., 2018).
Revised: CIFAR-10 with test accuracy of up to 92.37% (Hu et al., 2018; Sengupta et al., 2018).

Original: However, although SNNs compute with biologically realistic spikes, none of these models apply conductance averaging as is present in biological neurons, and they rely on BatchNorm-inspired layers.
Revised: However, although SNNs compute with biologically realistic spikes, none of these models apply conductance averaging as is present in biological neurons, and they rely on BatchNorm-inspired layers.

Original: Huh &
Revised: Huh &

Original: Sejnowski (2018) published a method for supervised training in SNNs with a conductance-based rule, but they did not test the model on computer vision datasets.
Revised: Sejnowski (2018) published a method for supervised training in SNNs with a conductance-based rule, but they did not test the model on computer vision datasets.

Original: Diehl & Cook (2015) simulated
Revised: Diehl & Cook (2015) simulated

Original: SNNs with conductances and excitatory-inhibitory neurons on MNIST, reaching 95% accuracy on
Revised: SNNs with conductances and excitatory-inhibitory neurons on MNIST, reaching 95% accuracy on

Original: MNIST with an unsupervised technique.
Revised: MNIST with an unsupervised technique.

Original: Diehl et al.
Revised: Diehl et al.

Original: (2015) used spiking afﬁne networks, but with a renormalization scheme that divided all weights in a layer by the maximum positive activity to prevent unbounded spiking.
Revised: (2015) used spiking afﬁne networks, but with a renormalization scheme that divided all weights in a layer by the maximum positive activity to prevent unbounded spiking.

Original: This method is loosely reminiscent of but distinct from CWA.
Revised: This method is loosely reminiscent of but distinct from CWA.

Original: It achieved 99.15% test accuracy on MNIST in a convolutional SNN, similar to the 99.02% reported below.
Revised: It achieved 99.15% test accuracy on MNIST in a convolutional SNN, similar to the 99.02% reported below.

Original: Excitatory-Inhibitory Networks.
Revised: Excitatory-Inhibitory Networks.

Original: Few classiﬁcation results have been reported using neurons divided into excitatory and inhibitory groups.
Revised: Few classiﬁcation results have been reported using neurons divided into excitatory and inhibitory groups.

Original: Diehl & Cook (2015) has already been mentioned.
Revised: Diehl & Cook (2015) has already been mentioned.

Original: Delahunt & Kutz (2018) modeled the olfactory bulb of moths and applied it to MNIST with 75-85% accuracy.
Revised: Delahunt & Kutz (2018) modeled the olfactory bulb of moths and applied it to MNIST with 75-85% accuracy.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2017) introduced a method that modeled excitatory neurons as ReLU activated and inhibitory neurons as activated with a negative-valued saturating ReLU, attaining 94.22% on
Revised: (2017) introduced a method that modeled excitatory neurons as ReLU activated and inhibitory neurons as activated with a negative-valued saturating ReLU, attaining 94.22% on

Original: CIFAR-10 with networks containing both excitatory and inhibitory activations inside of an otherwise standard CNN; this result slightly exceeds that obtained with their model for excitatory-only networks, suggesting that there is value in using multiple neuron types.
Revised: CIFAR-10 with networks containing both excitatory and inhibitory activations inside of an otherwise standard CNN; this result slightly exceeds that obtained with their model for excitatory-only networks, suggesting that there is value in using multiple neuron types.

Original: Normalization and Regularization Methods.
Revised: Normalization and Regularization Methods.

Original: CWA keeps neural activity within a ﬁxed range independent of the synapse strengths.
Revised: CWA keeps neural activity within a ﬁxed range independent of the synapse strengths.

Original: Within deep learning, it is well known that large weights and activations can be a symptom of overﬁtting the training data.
Revised: Within deep learning, it is well known that large weights and activations can be a symptom of overﬁtting the training data.

Original: Techniques to control this include
Revised: Techniques to control this include

Original: L2-regularization (Ng, 2004) of weights ( i.e. , weight decay) and activities.
Revised: L2-regularization (Ng, 2004) of weights ( i.e. , weight decay) and activities.

Original: An older technique, the
Revised: An older technique, the

Original: Neocognitron, explicitly normalizes neural activities, inspired by the concept of complex vs. simple cells in the brain Fukushima et al.
Revised: Neocognitron, explicitly normalizes neural activities, inspired by the concept of complex vs. simple cells in the brain Fukushima et al.

Original: Dropout Srivastava et al.
Revised: Dropout Srivastava et al.

Original: (2014) was proposed as a method of balancing the dependence of a neuron over its inputs, which also helps to prevent large variations in weight and activation magnitude.
Revised: (2014) was proposed as a method of balancing the dependence of a neuron over its inputs, which also helps to prevent large variations in weight and activation magnitude.

Original: Krizhevsky et al.
Revised: Krizhevsky et al.

Original: (2012) introduced Local Response Normalization as a method of locally smoothing neural activity in the hidden layers of a CNN.
Revised: (2012) introduced Local Response Normalization as a method of locally smoothing neural activity in the hidden layers of a CNN.

Original: It was superseded by BatchNorm (Ioffe & Szegedy, 2015), which explicitly centers and rescales neural activities to a standard Gaussian.
Revised: It was superseded by BatchNorm (Ioffe & Szegedy, 2015), which explicitly centers and rescales neural activities to a standard Gaussian.

Original: Most state-of-the-art results in deep learning rely heavily on BatchNorm to keep neural activities well-conditioned and to improve the curvature of the gradients during learning.
Revised: Most state-of-the-art results in deep learning rely heavily on BatchNorm to keep neural activities well-conditioned and to improve the curvature of the gradients during learning.

Original: One hypothesis regarding conductance networks is that by imposing a ﬁnite range on the neural activities, CWA may play a role in the brain similar to that of BatchNorm within artiﬁcial neural networks.
Revised: One hypothesis regarding conductance networks is that by imposing a ﬁnite range on the neural activities, CWA may play a role in the brain similar to that of BatchNorm within artiﬁcial neural networks.

Original: An advantage of CWA over BatchNorm is that it does not require extra parameters and does not impose requirements on how the training data are sampled in order to train the network.
Revised: An advantage of CWA over BatchNorm is that it does not require extra parameters and does not impose requirements on how the training data are sampled in order to train the network.

Original: This last point is relevant since BatchNorm makes importance sampling difﬁcult, which has prevented its use, for example, within deep reinforcement learning, where priority sampling is commonly applied.
Revised: This last point is relevant since BatchNorm makes importance sampling difﬁcult, which has prevented its use, for example, within deep reinforcement learning, where priority sampling is commonly applied.

Original: CWA could be used to ﬁll the role of BatchNorm in these cases; as best as is currently known, the human brain is trained via reinforcement learning.
Revised: CWA could be used to ﬁll the role of BatchNorm in these cases; as best as is currently known, the human brain is trained via reinforcement learning.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In order to assess its experimental value, CWA can be used as a drop-in replacement for the classical afﬁne transformation inside of existing fully-connected and convolutional architectures.
Revised: In order to assess its experimental value, CWA can be used as a drop-in replacement for the classical afﬁne transformation inside of existing fully-connected and convolutional architectures.

Original: The resulting artiﬁcial neural networks and some variants have been tested on two image classiﬁcation tasks,
Revised: The resulting artiﬁcial neural networks and some variants have been tested on two image classiﬁcation tasks,

Original: MNIST and CIFAR-10.The architectures, datasets, and results are described in this section.
Revised: MNIST and CIFAR-10.The architectures, datasets, and results are described in this section.

Original: 4. C OMPUTER V ISION D ATASETS
Revised: 4. C OMPUTER V ISION D ATASETS

Original: MNIST (LeCun & Cortes, 2010) consists of a training set of 60,000 images of handwritten digits from 0 to 9, with a test set of 10,000 images.
Revised: MNIST (LeCun & Cortes, 2010) consists of a training set of 60,000 images of handwritten digits from 0 to 9, with a test set of 10,000 images.

Original: All images are 28 x 28 pixels in size, and the dataset is balanced among the ten digits, so that each digit has 6,000 labeled training examples and 1, labeled test examples.
Revised: All images are 28 x 28 pixels in size, and the dataset is balanced among the ten digits, so that each digit has 6,000 labeled training examples and 1, labeled test examples.

Original: CIFAR-10 (Krizhevsky et al., 2009) contains 50,000 images for training and 10,000 for test, where the images are divided into 10 balanced classes such as airplanes , cars , trucks , and dogs .
Revised: CIFAR-10 (Krizhevsky et al., 2009) contains 50,000 images for training and 10,000 for test, where the images are divided into 10 balanced classes such as airplanes , cars , trucks , and dogs .

Original: Each image in CIFAR10 has 32 x 32 x 3 pixels.
Revised: Each image in CIFAR10 has 32 x 32 x 3 pixels.

Original: In the experiments below, the only data augmentation applied to CIFAR-10 was random shifting by up to 4 pixels followed by cropping, and random horizontal ﬂipping of the image.
Revised: In the experiments below, the only data augmentation applied to CIFAR-10 was random shifting by up to 4 pixels followed by cropping, and random horizontal ﬂipping of the image.

Original: This augmentation regime is sufﬁcient to train common architectures to over 90% accuracy.
Revised: This augmentation regime is sufﬁcient to train common architectures to over 90% accuracy.

Original: Preprocessing.
Revised: Preprocessing.

Original: Ordinarily, images are preprocessed before presentation to a neural network.
Revised: Ordinarily, images are preprocessed before presentation to a neural network.

Original: This processing typically includes some form of whitening or contrast normalization, which makes the images more consistent and easier to classify.
Revised: This processing typically includes some form of whitening or contrast normalization, which makes the images more consistent and easier to classify.

Original: In the experiments below, only minimal preprocessing was performed.
Revised: In the experiments below, only minimal preprocessing was performed.

Original: For CWA, the color channel values were scaled to lie between [ − 0 . 3 , 1] and then clipped to [0 , 1] .
Revised: For CWA, the color channel values were scaled to lie between [ − 0 . 3 , 1] and then clipped to [0 , 1] .

Original: Much like in Equation 4, this clipping was done to promote sparsity and had a positive effect on learning speed and generalization accuracy.
Revised: Much like in Equation 4, this clipping was done to promote sparsity and had a positive effect on learning speed and generalization accuracy.

Original: For afﬁne networks, the color channels were mean-centered and variance-scaled.
Revised: For afﬁne networks, the color channels were mean-centered and variance-scaled.

Original: This whitening could not be used for CWA because CWA assumes that the inputs lie in the interval [0 , 1] .
Revised: This whitening could not be used for CWA because CWA assumes that the inputs lie in the interval [0 , 1] .

Original: Although these datasets are not sufﬁcient to establish state-of-the-art performance for CWA, they do comprise a strong bank of tests to establish that CWA can solve computer vision tasks at the level of the state-of-the-art for afﬁne networks from just 2-3 years prior.
Revised: Although these datasets are not sufﬁcient to establish state-of-the-art performance for CWA, they do comprise a strong bank of tests to establish that CWA can solve computer vision tasks at the level of the state-of-the-art for afﬁne networks from just 2-3 years prior.

Original: 4. A RCHITECTURES
Revised: 4. A RCHITECTURES

Original: One of the key advances in neural networks in the past decade has been the development of novel architectures, including the layout of artiﬁcial neurons and the introduction of components to address information-processing concerns.
Revised: One of the key advances in neural networks in the past decade has been the development of novel architectures, including the layout of artiﬁcial neurons and the introduction of components to address information-processing concerns.

Original: For CWA, three common architectures have been adapted for experiments on the datasets.
Revised: For CWA, three common architectures have been adapted for experiments on the datasets.

Original: Fully Connected Network (FCN).
Revised: Fully Connected Network (FCN).

Original: FCNs consist of several layers of neurons with feed-forward computation.
Revised: FCNs consist of several layers of neurons with feed-forward computation.

Original: For CWA, the experiments partition each layer into excitatory and inhibitory neurons, with half the neurons in the layer being assigned to each category.
Revised: For CWA, the experiments partition each layer into excitatory and inhibitory neurons, with half the neurons in the layer being assigned to each category.

Original: The neurons in the input layer were doubled, with one excitatory copy and one inhibitory.
Revised: The neurons in the input layer were doubled, with one excitatory copy and one inhibitory.

Original: Convolutional Neural Network (CNN).
Revised: Convolutional Neural Network (CNN).

Original: Convolutional neural networks adapt a feed-forward network to incorporate local connections (receptive ﬁelds) and shared weights.
Revised: Convolutional neural networks adapt a feed-forward network to incorporate local connections (receptive ﬁelds) and shared weights.

Original: The experiments below use a sequence of four or ﬁve convolutional layers with stride 2 and no pooling.
Revised: The experiments below use a sequence of four or ﬁve convolutional layers with stride 2 and no pooling.

Original: To implement
Revised: To implement

Original: CWA-CNNs, the simplest adaptation is to assign each presynaptic channel a valence ( i.e. , zerocurrent potential) in the context of the receptive ﬁeld.
Revised: CWA-CNNs, the simplest adaptation is to assign each presynaptic channel a valence ( i.e. , zerocurrent potential) in the context of the receptive ﬁeld.

Original: For the experiments, half of the channels were considered excitatory and half inhibitory.
Revised: For the experiments, half of the channels were considered excitatory and half inhibitory.

Original: To adapt the input image, the input channels were doubled from three to six, with two channels for each color, one excitatory and one inhibitory.
Revised: To adapt the input image, the input channels were doubled from three to six, with two channels for each color, one excitatory and one inhibitory.

Original: Using tensor notation, for the standard convolutional ﬁlter one writes
Revised: Using tensor notation, for the standard convolutional ﬁlter one writes

Original: [Equation]
Revised: [Equation]

Original: which is adapted to the conductance network as
Revised: which is adapted to the conductance network as

Original: [Equation]
Revised: [Equation]

Original: where G ijxy =
Revised: where G ijxy =

Original: h ( g ijxy ) is the max conductance parameter, ⊗ indicates componentwise multiplication (also known as the Hadamard product) and
Revised: h ( g ijxy ) is the max conductance parameter, ⊗ indicates componentwise multiplication (also known as the Hadamard product) and

Original: [Equation]
Revised: [Equation]

Original: where J is the number of presynaptic channels to layer i , i.e. , half the channels are excitatory and half inhibitory.
Revised: where J is the number of presynaptic channels to layer i , i.e. , half the channels are excitatory and half inhibitory.

Original: Here G ⊗ e is modeled rather than p ⊗ e to reﬂect the concept that the neuron valence is determined by the receptive ﬁeld, but the two concepts are of equivalent signiﬁcance given the deﬁnition of e above.
Revised: Here G ⊗ e is modeled rather than p ⊗ e to reﬂect the concept that the neuron valence is determined by the receptive ﬁeld, but the two concepts are of equivalent signiﬁcance given the deﬁnition of e above.

Original: Residual Networks (ResNet).
Revised: Residual Networks (ResNet).

Original: Residual Networks (He et al., 2016) are a variant of convolutional neural networks characterized by blocks of successive convolutions with shortcut connections.
Revised: Residual Networks (He et al., 2016) are a variant of convolutional neural networks characterized by blocks of successive convolutions with shortcut connections.

Original: This paper uses the ResNet-18 architecture without BatchNorm but with CWA convolutions as in
Revised: This paper uses the ResNet-18 architecture without BatchNorm but with CWA convolutions as in

Original: Equation 6 and activated according to Equation 4.
Revised: Equation 6 and activated according to Equation 4.

Original: 4. T RAINING R ESULTSMNIST
Revised: 4. T RAINING R ESULTS

Original: CIFAR-

Original: CWA A FFINE + BN CWA
Revised: Each of the three architectures was tested on both datasets for both afﬁne networks and CWA.

Original: CWA+BN A FFINE N O BN A FFINE + BN FCN 98.
Revised: For clarity, the afﬁne networks used ReLU activation (Nair & Hinton, 2010) and were initialized with

Original: [Table]
Revised: [Table]

Original: Each of the three architectures was tested on both datasets for both afﬁne networks and CWA.

Original: For clarity, the afﬁne networks used ReLU activation (Nair & Hinton, 2010) and were initialized with

Original: Glorot initialization (Glorot & Bengio, 2010).
Revised: Glorot initialization (Glorot & Bengio, 2010).

Original: Two variants of afﬁne networks were tested.
Revised: Two variants of afﬁne networks were tested.

Original: The ﬁrst variant (Afﬁne, No BN) took the same unwhitened, scaled inputs as CWA and did not use batch norm.
Revised: The ﬁrst variant (Afﬁne, No BN) took the same unwhitened, scaled inputs as CWA and did not use batch norm.

Original: In the second variant (Afﬁne + BN), image inputs were whitened and each ReLU activation was preceded by an application of BatchNorm.
Revised: In the second variant (Afﬁne + BN), image inputs were whitened and each ReLU activation was preceded by an application of BatchNorm.

Original: The purpose of training afﬁne networks is to obtain meaningful baselines for comparison.
Revised: The purpose of training afﬁne networks is to obtain meaningful baselines for comparison.

Original: In one case, CWA was tested with BatchNorm applied similarly (CWA+BN)
Revised: In one case, CWA was tested with BatchNorm applied similarly (CWA+BN)

Original: The training procedures are described below, followed by the results.
Revised: The training procedures are described below, followed by the results.

Original: Initialization.
Revised: Initialization.

Original: The g ij ’s and g ijxy ’s comprise all of the parameters in a conductance network.
Revised: The g ij ’s and g ijxy ’s comprise all of the parameters in a conductance network.

Original: These parameters were Glorot-initialized, so that the initial G i,j parameters varied tightly around 1 .
Revised: These parameters were Glorot-initialized, so that the initial G i,j parameters varied tightly around 1 .

Original: Unlike with standard neural networks, CWA parameters can be initialized to zero without preventing learning, although we did not do so.
Revised: Unlike with standard neural networks, CWA parameters can be initialized to zero without preventing learning, although we did not do so.

Original: To enforce the nonnegativity of G i,j , h ( g ij )
Revised: To enforce the nonnegativity of G i,j , h ( g ij )

Original: =
Revised: =

Original: (1 + g ij )
Revised: (1 + g ij )

Original: + was used in
Revised: + was used in

Original: Equation 3.
Revised: Equation 3.

Original: Although this function is discontinuous, by initializing the g ij close to zero, a wide range is available for optimization, and the linear scaling of the gradient proved to be advantageous from a numerical perspective.
Revised: Although this function is discontinuous, by initializing the g ij close to zero, a wide range is available for optimization, and the linear scaling of the gradient proved to be advantageous from a numerical perspective.

Original: By contrast, h = exp could have been used, but it caused overﬂow at later stages of training in some cases during preliminary experiments.
Revised: By contrast, h = exp could have been used, but it caused overﬂow at later stages of training in some cases during preliminary experiments.

Original: Bias Term.
Revised: Bias Term.

Original: The bias conductance G r = h ( g r ) and its valence e r , representing the neuron’s own zero-current potential, were ﬁxed and not adapted during training.
Revised: The bias conductance G r = h ( g r ) and its valence e r , representing the neuron’s own zero-current potential, were ﬁxed and not adapted during training.

Original: Speciﬁcally, G r = 1 was used, and e r = − 0 .
Revised: Speciﬁcally, G r = 1 was used, and e r = − 0 .

Original: 4 was used for FCN but e r = 0 for CNN and ResNet.
Revised: 4 was used for FCN but e r = 0 for CNN and ResNet.

Original: By ﬁxing these values, the denominator in Equations 3 and 6 could never produce division by zero.
Revised: By ﬁxing these values, the denominator in Equations 3 and 6 could never produce division by zero.

Original: The inhibitory bias for FCN was determined experimentally in order to produce a tendency towards not ﬁring in the zero-input state.
Revised: The inhibitory bias for FCN was determined experimentally in order to produce a tendency towards not ﬁring in the zero-input state.

Original: In convolutional architectures, setting e r to any value other than zero caused training to fail in preliminary experiments for unknown reasons.
Revised: In convolutional architectures, setting e r to any value other than zero caused training to fail in preliminary experiments for unknown reasons.

Original: Categorical Output.
Revised: Categorical Output.

Original: The output of an afﬁne network on categorical networks is traditionally determined by applying a softmax activation, P ( c i ) ∝ exp( z i ) , in place of the standard activation.
Revised: The output of an afﬁne network on categorical networks is traditionally determined by applying a softmax activation, P ( c i ) ∝ exp( z i ) , in place of the standard activation.

Original: Biological networks do not seem to have categorical outputs of this form.
Revised: Biological networks do not seem to have categorical outputs of this form.

Original: Thus in order to compute a categorical output, a ﬁnal afﬁne layer with softmax activation was added as the top layer of each
Revised: Thus in order to compute a categorical output, a ﬁnal afﬁne layer with softmax activation was added as the top layer of each

Original: CWA architecture to extract the categories from the CWA network.
Revised: CWA architecture to extract the categories from the CWA network.

Original: Parameter Fitting.
Revised: Parameter Fitting.

Original: All networks, both CWA and afﬁne networks, were trained with backprop using mini-batch stochastic gradient descent with the Adam optimizer (Kingma & Ba, 2014) to control per-parameter learning rates ( β 1 = 0 .
Revised: All networks, both CWA and afﬁne networks, were trained with backprop using mini-batch stochastic gradient descent with the Adam optimizer (Kingma & Ba, 2014) to control per-parameter learning rates ( β 1 = 0 .

Original: 9 , β 2 = 0 . 99 ).
Revised: 9 , β 2 = 0 . 99 ).

Original: Lower initial learning rates ( η = 1 e − 5 ) were found to be necessary in most cases for CWA, and the resulting parameter learning was noticeably slower than for afﬁne networks, which were trained with initial η = 0 .
Revised: Lower initial learning rates ( η = 1 e − 5 ) were found to be necessary in most cases for CWA, and the resulting parameter learning was noticeably slower than for afﬁne networks, which were trained with initial η = 0 .

Original: Backprop was used purely as a mathematical method of demonstrating that CWA networks do indeed have the capacity to ﬁt the data.
Revised: Backprop was used purely as a mathematical method of demonstrating that CWA networks do indeed have the capacity to ﬁt the data.

Original: Training Procedures.
Revised: Training Procedures.

Original: Each network was trained until it had ﬁt the training set to high accuracy.
Revised: Each network was trained until it had ﬁt the training set to high accuracy.

Original: In all cases the training accuracy was > 99 . 4% .
Revised: In all cases the training accuracy was > 99 . 4% .

Original: At each epoch of training, every training image was presented to the network in minibatches.
Revised: At each epoch of training, every training image was presented to the network in minibatches.

Original: At the end of each epoch, the training loss was computed, and if 25 epochs elapsed with no improvement in training loss for CWA, the learning rate was decreased by a factor of 10 .
Revised: At the end of each epoch, the training loss was computed, and if 25 epochs elapsed with no improvement in training loss for CWA, the learning rate was decreased by a factor of 10 .

Original: For afﬁne networks, only 5 epochs without decrease were allowed.
Revised: For afﬁne networks, only 5 epochs without decrease were allowed.

Original: The test accuracy was computed after each epoch, and the lowest test accuracy during training is reported for each network.
Revised: The test accuracy was computed after each epoch, and the lowest test accuracy during training is reported for each network.

Original: In general, test accuracy did not substantially decrease at any point during training.
Revised: In general, test accuracy did not substantially decrease at any point during training.

Original: No validation sets were used; all networks were trained on all training data.
Revised: No validation sets were used; all networks were trained on all training data.

Original: Network Sizes.
Revised: Network Sizes.

Original: The weight-bearing layers were of the same size for both CWA and afﬁne networks in all cases, but varied according to dataset.
Revised: The weight-bearing layers were of the same size for both CWA and afﬁne networks in all cases, but varied according to dataset.

Original: For MNIST, the FCN had two hidden layers of 2 ,neurons each.
Revised: For MNIST, the FCN had two hidden layers of 2 ,neurons each.

Original: For CIFAR-10, a ﬁve-layer FCN with three hidden layers of 2000 , 1000 , 500 was used.
Revised: For CIFAR-10, a ﬁve-layer FCN with three hidden layers of 2000 , 1000 , 500 was used.

Original: The CNN for MNIST had four convolutional layers with 32, 64, 128, and 256 channels, respectively.
Revised: The CNN for MNIST had four convolutional layers with 32, 64, 128, and 256 channels, respectively.

Original: The CNN for CIFAR-10 added a ﬁfth convolutional layer with 512 channels.
Revised: The CNN for CIFAR-10 added a ﬁfth convolutional layer with 512 channels.

Original: ResNet-18 consists of four blocks of two convolutions each.
Revised: ResNet-18 consists of four blocks of two convolutions each.

Original: The initial input was projected to 64 channels, and the four blocks internally used 64, 128, 256, and 512 channels, respectively.
Revised: The initial input was projected to 64 channels, and the four blocks internally used 64, 128, 256, and 512 channels, respectively.

Original: The ResNet-18 architecture was not changed across tasks.
Revised: The ResNet-18 architecture was not changed across tasks.

Original: Results.
Revised: Results.

Original: The results are shown in Table 4.3.
Revised: The results are shown in Table ?? .

Original: Focusing ﬁrst on the fully connected architecture, CWA is seen to be competitive with afﬁne networks with BatchNorm.
Revised: Focusing ﬁrst on the fully connected architecture, CWA is seen to be competitive with afﬁne networks with BatchNorm.

Original: On MNIST, the two are nearly equal at 98 .
Revised: On MNIST, the two are nearly equal at 98 .

Original: 65% and 98 .
Revised: 65% and 98 .

Original: 72% , and on CIFAR-10 they differ by less than 2% at 61 .
Revised: 72% , and on CIFAR-10 they differ by less than 2% at 61 .

Original: 98% and 63 .
Revised: 98% and 63 .

Original: Adding BatchNorm to CWA does however improve performance to 64 .
Revised: Adding BatchNorm to CWA does however improve performance to 64 .

Original: There are many opportunities to close the gap between CWA and Afﬁne networks.
Revised: There are many opportunities to close the gap between CWA and Afﬁne networks.

Original: The input and output procedures are not biologically driven and may not be optimal for CWA.
Revised: The input and output procedures are not biologically driven and may not be optimal for CWA.

Original: For example, the afﬁne networks have the beneﬁt of using whitened inputs, whereas the CWA inputs are merely scaled and clipped.
Revised: For example, the afﬁne networks have the beneﬁt of using whitened inputs, whereas the CWA inputs are merely scaled and clipped.

Original: Different connectivity patterns between excitatory and inhibitory inputs may also help as well.
Revised: Different connectivity patterns between excitatory and inhibitory inputs may also help as well.

Original: In the case of the CNN and ResNet-18, there is a greater distance between the results for CWA and afﬁne networks, but the results for CWA on CIFAR-10 are still comparable to published results from approximately 2012 to 2014 (see e.g. http://rodrigob.github.io/are_we_there_ yet/build/ ).
Revised: In the case of the CNN and ResNet-18, there is a greater distance between the results for CWA and afﬁne networks, but the results for CWA on CIFAR-10 are still comparable to published results from approximately 2012 to 2014 (see e.g. http://rodrigob.github.io/are_we_there_ yet/build/ ).

Original: It is also noteworthy, that without BatchNorm, afﬁne networks underperform CWA.
Revised: It is also noteworthy, that without BatchNorm, afﬁne networks underperform CWA.

Original: One reason why the convolutional version of CWA performs worse might have to do with the low number of incoming connections.
Revised: One reason why the convolutional version of CWA performs worse might have to do with the low number of incoming connections.

Original: The ﬁrst convolutional kernel in CNN has merely 6 x 5 x 5 =connections for each neuron in the second layer.
Revised: The ﬁrst convolutional kernel in CNN has merely 6 x 5 x 5 =connections for each neuron in the second layer.

Original: This number should be compared with 32 xx 6 = 6,144 connections for each neuron in the second layer of the FCN and tens of thousands of connections neurons in the brain.
Revised: This number should be compared with 32 xx 6 = 6,144 connections for each neuron in the second layer of the FCN and tens of thousands of connections neurons in the brain.

Original: Higher connectivity may be important to good performance with
Revised: Higher connectivity may be important to good performance with

Original: CWA, because it allows the denominator of Equation 3 to depend on many more input features.
Revised: CWA, because it allows the denominator of Equation 3 to depend on many more input features.

Original: When there are fewer connections, a handful of input features can dominate the denominator, reducing generalization performance.
Revised: When there are fewer connections, a handful of input features can dominate the denominator, reducing generalization performance.

Original: Wider convolutional kernels might help, but are impractical on small images such as those of CIFAR and MNIST.
Revised: Wider convolutional kernels might help, but are impractical on small images such as those of CIFAR and MNIST.

Original: Better performance might be possible on datasets of larger images.
Revised: Better performance might be possible on datasets of larger images.

Original: It must be added that as an architecture, CNNs uses weight sharing in order to achieve economy with respect to data and memory.
Revised: It must be added that as an architecture, CNNs uses weight sharing in order to achieve economy with respect to data and memory.

Original: Biological neural networks are organized differently, and it may be that a new architectural paradigm is needed for CWA.
Revised: Biological neural networks are organized differently, and it may be that a new architectural paradigm is needed for CWA.

Original: Regardless of any deﬁciencies, the results above demonstrate that a network with conductance averaging and excitatory-inhibitory neurons such as CWA can solve real problems.
Revised: Regardless of any deﬁciencies, the results above demonstrate that a network with conductance averaging and excitatory-inhibitory neurons such as CWA can solve real problems.

Original: To our knowledge, this is the ﬁrst time that these two biological features have been successfully combined inside of an artiﬁcial neural network.
Revised: To our knowledge, this is the ﬁrst time that these two biological features have been successfully combined inside of an artiﬁcial neural network.

Original: 4. CWA V ARIANTS : A BS N ORM AND L ATERAL I NHIBITION
Revised: 4. CWA V ARIANTS : A BS N ORM AND L ATERAL I NHIBITION

Original: Networks with excitatory and inhibitory neurons and strictly positive weights are a consequence of
Revised: Networks with excitatory and inhibitory neurons and strictly positive weights are a consequence of

Original: Equation 1, but there is a generalization of conductance averaging that can be applied to afﬁne nets as an ablation study on CWA.
Revised: Equation 1, but there is a generalization of conductance averaging that can be applied to afﬁne nets as an ablation study on CWA.

Original: The basic idea is to make the valence
Revised: The basic idea is to make the valence

Original: e j = ± 1 a property of the synapse ( e i,j ) rather than the neuron and then assign W i,j = G i,j e i,j so that | W i,j | =
Revised: e j = ± 1 a property of the synapse ( e i,j ) rather than the neuron and then assign W i,j = G i,j e i,j so that | W i,j | =

Original: G i,j .
Revised: G i,j .

Original: Then Equationbecomes
Revised: Then Equationbecomes

Original: [Equation]
Revised: [Equation]

Original: which is just Equation 2 with divisive norming based on the absolute value of the weights.
Revised: which is just Equation 2 with divisive norming based on the absolute value of the weights.

Original: This equation will be called AbsNorm.
Revised: This equation will be called AbsNorm.

Original: As with CWA, it forces z i ∈ [ − 1 , 1] , so that applying a ReLU afterwards results in p i ∈ [0 , 1] .
Revised: As with CWA, it forces z i ∈ [ − 1 , 1] , so that applying a ReLU afterwards results in p i ∈ [0 , 1] .

Original: AbsNorm characterizes the two main differences between CWA and afﬁne nets, namely, the divisive averaging based on total input activity and the pattern in which outgoing connections from a neuron all share the same valence.
Revised: AbsNorm characterizes the two main differences between CWA and afﬁne nets, namely, the divisive averaging based on total input activity and the pattern in which outgoing connections from a neuron all share the same valence.

Original: Interestingly, AbsNorm failed to perform as well as CWA in preliminary experiments with the FCN architecture on CIFAR-10; these results are shown in Table 4.4.
Revised: Interestingly, AbsNorm failed to perform as well as CWA in preliminary experiments with the FCN architecture on CIFAR-10; these results are shown in Table ?? .

Original: The difference between CWA and AbsNorm is substantial, and was consistently worth about 2% test accuracy in four separate trials, the highest of which is shown for each in the table.
Revised: The difference between CWA and AbsNorm is substantial, and was consistently worth about 2% test accuracy in four separate trials, the highest of which is shown for each in the table.

Original: This result suggests that there is value in separating the neurons into distinct valence classes.
Revised: This result suggests that there is value in separating the neurons into distinct valence classes.

Original: Distinct valences delineate of features as having an inherently positive or negative signiﬁcance, though it is not clear why this yields better performance at this time.[Table]
Revised: Distinct valences delineate of features as having an inherently positive or negative signiﬁcance, though it is not clear why this yields better performance at this time.

Original: T ABLE 2: CWA V ARIANTS , FCN A CCURACY ON CIFAR-

Original: [Table]

Original: One major area in which the architectures tested in this paper are not biologically faithful is the connectivity pattern among different neuron types.
Revised: One major area in which the architectures tested in this paper are not biologically faithful is the connectivity pattern among different neuron types.

Original: This is an interesting aspect for future work.
Revised: This is an interesting aspect for future work.

Original: As a ﬁrst step in this direction, extra inhibitory neurons were added to each layer of CWA and  activated recurrently in a temporal rollout, simulating the effect of lateral inhibitory interneurons, which potentially regulate the overall activity level within cortical regions.
Revised: As a ﬁrst step in this direction, extra inhibitory neurons were added to each layer of CWA and [Table] activated recurrently in a temporal rollout, simulating the effect of lateral inhibitory interneurons, which potentially regulate the overall activity level within cortical regions.

Original: As seen in Table 4.4, CWA with this addition did achieve the highest accuracy among CWA variants tested, but only marginally so.
Revised: As seen in Table ?? , CWA with this addition did achieve the highest accuracy among CWA variants tested, but only marginally so.

Original: More work is needed to establish the effect of different connectivity patterns.
Revised: More work is needed to establish the effect of different connectivity patterns.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Limitations & Future Work.
Revised: Limitations & Future Work.

Original: There are several ways to improve CWA.
Revised: There are several ways to improve CWA.

Original: Most prominently, the decision to split layers into half excitatory and half inhibitory neurons connected in a feed-forward fashion was somewhat arbitrary and does not reﬂect biology.
Revised: Most prominently, the decision to split layers into half excitatory and half inhibitory neurons connected in a feed-forward fashion was somewhat arbitrary and does not reﬂect biology.

Original: It would be interesting to approximate biological valence ratios and connectivity patterns to understand better how these features contribute to information processing in neural networks.
Revised: It would be interesting to approximate biological valence ratios and connectivity patterns to understand better how these features contribute to information processing in neural networks.

Original: Realistic connectivity patterns would require the use of a recurrent activation pattern rather than feed-forward.
Revised: Realistic connectivity patterns would require the use of a recurrent activation pattern rather than feed-forward.

Original: Although substantial work has been done simulating the dynamics of spiking neural networks, less research has been directed towards discrete time slice approximations as used in recurrent neural networks (RNNs).
Revised: Although substantial work has been done simulating the dynamics of spiking neural networks, less research has been directed towards discrete time slice approximations as used in recurrent neural networks (RNNs).

Original: The same principles of this paper could be applied to that case as well.
Revised: The same principles of this paper could be applied to that case as well.

Original: Separately, it is notable that the fully connected architecture obtained closer results to the afﬁne baseline than convolutional architectures.
Revised: Separately, it is notable that the fully connected architecture obtained closer results to the afﬁne baseline than convolutional architectures.

Original: Convolutional neurons in deep learning generally have fewer incoming connections than biological neurons do, and in may be that high connectivity is needed for CWA to work well.
Revised: Convolutional neurons in deep learning generally have fewer incoming connections than biological neurons do, and in may be that high connectivity is needed for CWA to work well.

Original: This issue needs further exploration.
Revised: This issue needs further exploration.

Original: Finally, parameter ﬁtting with CWA was based on backprop.
Revised: Finally, parameter ﬁtting with CWA was based on backprop.

Original: There is no conclusive evidence that animal brains adapt their conductances using backprop, and it would be worthwhile to explore alternatives for parameter ﬁtting that do not use backprop but work well with CWA.
Revised: There is no conclusive evidence that animal brains adapt their conductances using backprop, and it would be worthwhile to explore alternatives for parameter ﬁtting that do not use backprop but work well with CWA.

Original: Applications to Deep Learning.
Revised: Applications to Deep Learning.

Original: CWA inherently implements features that have been found to be useful in deep learning, such as sparsity and limiting the output range of neurons.
Revised: CWA inherently implements features that have been found to be useful in deep learning, such as sparsity and limiting the output range of neurons.

Original: CWA does so by dynamically scaling the output to marginalize out the level of input activity.
Revised: CWA does so by dynamically scaling the output to marginalize out the level of input activity.

Original: It would be of interest to understand how this dynamic scaling affects other properties of afﬁne networks, such as susceptibility to adversarial images.
Revised: It would be of interest to understand how this dynamic scaling affects other properties of afﬁne networks, such as susceptibility to adversarial images.

Original: Furthermore, the network architectures tested in this paper as well as the training procedures were developed for afﬁne networks.
Revised: Furthermore, the network architectures tested in this paper as well as the training procedures were developed for afﬁne networks.

Original: It may be that with specially adapted training procedures or architectures, CWA could still achieve state-of-the-art results.
Revised: It may be that with specially adapted training procedures or architectures, CWA could still achieve state-of-the-art results.

Original: Applications to Neuroscience.
Revised: Applications to Neuroscience.

Original: In contrast to many biologically motivated neuronal simulations, the
Revised: In contrast to many biologically motivated neuronal simulations, the

Original: CWA experiments in this paper solve serious computer vision problems, suggesting that the value of neuroscience research can be illustrated by application to deep learning problems.
Revised: CWA experiments in this paper solve serious computer vision problems, suggesting that the value of neuroscience research can be illustrated by application to deep learning problems.

Original: However, nonbiological adaptations were required to process the inputs and outputs of the network and to control parameter adaptation.
Revised: However, nonbiological adaptations were required to process the inputs and outputs of the network and to control parameter adaptation.

Original: It would be interesting to understand the extent to which more biologically accurate settings can still obtain similar results.
Revised: It would be interesting to understand the extent to which more biologically accurate settings can still obtain similar results.

Original: The experimental setting of this paper also offers a miniature lab in order to test questions such as whether multiple neuron valences promote better learning, or how the balance and connection among these valence types affect information processing and learning.
Revised: The experimental setting of this paper also offers a miniature lab in order to test questions such as whether multiple neuron valences promote better learning, or how the balance and connection among these valence types affect information processing and learning.

Original: Although such experiments could not conclude anything about what biological systems actually do, they could be invaluable in identifying possibilities for consideration.
Revised: Although such experiments could not conclude anything about what biological systems actually do, they could be invaluable in identifying possibilities for consideration.

Original: To conclude, CWA implements a biologically accurate understanding of neuronal activity and for the ﬁrst time achieves competitive performance at computer vision datasets in a network with conductance averaging and excitatory-inhibitory neurons.
Revised: To conclude, CWA implements a biologically accurate understanding of neuronal activity and for the ﬁrst time achieves competitive performance at computer vision datasets in a network with conductance averaging and excitatory-inhibitory neurons.

Original: This research provides a pattern for identifying and abstracting basic biophysical knowledge to demonstrate relevance to machine learning.
Revised: This research provides a pattern for identifying and abstracting basic biophysical knowledge to demonstrate relevance to machine learning.

Revised:  S UPPLEMENTARY M ATERIAL 6. C ODE I MPLEMENTING CWA

Revised: The following code implements a fully connected CWA layer in python as a PyTorch module.

Revised: As can be seen, CWA requires two matrix multiplies instead of the usual one.

Revised: class CWALayer(torch.nn.

Revised: Module): def __init__(self, neurons_in, neurons_out): super(CWALayer, self).__init__() self.neurons_in = neurons_in self.neurons_out = neurons_out self.gs

Revised: = torch.nn.

Revised: Parameter( torch.zeros(neurons_in, neurons_out) ) torch.nn.init.xavier_normal_(self.gs) self.es=torch.nn.

Revised: Parameter( torch.zeros(neurons_in).fill_(1.0), requires_grad=False ) self.es[:neurons_in//2] = -1.

Revised: def fix(self, gs): return torch.clamp(1+gs, 0.0, float(’inf’))

Revised: def forward(self, ps):

Revised: Gs = self.fix(self.gs) normalizer =

Revised: 1 + torch.matmul(ps,Gs) numerator =

Revised: -0.4 + torch.matmul(ps*self.es.unsqueeze(0), Gs) activity =

Revised: ( numerator / torch.clamp(normalizer, 1e-6, float(’inf’)) )

Revised: return activity

Revised: The following code implements a convolutional CWA layer.

Revised: As is seen below, implementing CWA requires two convolutions instead of one, but is otherwise similar to the afﬁne version.

Revised: class ConvCWALayer(torch.nn.

Revised: Module): def __

Revised: init

Revised: self

Revised: in

Revised: _channels,

Revised: out

Revised: _channels, kernel_size, stride=1, padding=0): super(

Revised: ConvCWALayer

Revised: , self).__init__() self.gs

Revised: = torch.nn.

Revised: Parameter nn nn requires grad padding padding grest nn def fix(self, gs): return np def forward(self, ps):

Revised: Gs = self.fix(self.gs) normalizer =

Revised: 1.0 + torch.nn.functional.conv2d( ps, Gs, stride=self.stride, padding=self.padding ) numerator = torch.nn.functional.conv2d( ps, Gs*self.es.unsqueeze(0), stride=self.stride, padding=self.padding ) return ( numerator / torch.clamp(normalizer, 1e-6, float(’inf’)) )