Original: L EARNING TO LIVE WITH D ALE ’ S PRINCIPLE : ANN S WITH SEPARATE EXCITATORY AND INHIBITORY UNITS
Revised: L EARNING TO LIVE WITH D ALE ’ S PRINCIPLE : ANN S WITH SEPARATE EXCITATORY AND INHIBITORY UNITS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The units in artiﬁcial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research.
Revised: The units in artiﬁcial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research.

Original: However, there are many important differences between ANN units and real neurons.
Revised: However, there are many important differences between ANN units and real neurons.

Original: One of the most notable is the absence of Dale’s principle which ensures that biological neurons are either exclusively excitatory or inhibitory.
Revised: One of the most notable is the absence of Dale’s principle, which ensures that biological neurons are either exclusively excitatory or inhibitory.

Original: Dale’s principle is typically left out of ANNs because its inclusion impairs learning.
Revised: Dale’s principle is typically left out of ANNs because its inclusion impairs learning.

Original: This is prob- lematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated realistic tasks.
Revised: This is problematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated, realistic tasks.

Original: Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop
Revised: Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop

Original: ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs
Revised: ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs.

Original: We call these networks Dale’s ANNs DANNs
Revised: We call these networks Dale’s ANNs (DANNs).

Original: We present two insights that enable DANNs to learn well: (1) DANNs are related to normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix.
Revised: We present two insights that enable DANNs to learn well: (1) DANNs are related to normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix.

Original: These results demonstrate how ANNs that respect Dale’s principle can be built without sacriﬁcing learning performance, which is important for future work using ANNs as models of the brain.
Revised: These results demonstrate how ANNs that respect Dale’s principle can be built without sacriﬁcing learning performance which is important for future work using ANNs as models of the brain.

Original: The resultsalso may  have interesting implications for how inhibitory plasticity in the real brain operates.
Revised: The results may also have interesting implications for how inhibitory plasticity in the real brain operates.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, artiﬁcial neural networks (ANNs) have been increasingly used in neuroscience research for modelling the brain at the algorithmic and computational level (Richards et al., 2019;
Revised: In recent years, artiﬁcial neural networks (ANNs) have been increasingly used in neuroscience research for modelling the brain at the algorithmic and computational level (Richards et al., 2019;

Original: Kietzmann et al., 2018; Yamins & DiCarlo, 2016).
Revised: Kietzmann et al., 2018; Yamins & DiCarlo, 2016).

Original: They have been used for exploring the structure of representations in the brain, the learning algorithms of the brain, and the behavioral patterns of humans and animals (Bartunov et al., 2018; Donhauser & Baillet, 2020; Michaels et al., 2019;
Revised: They have been used for exploring the structure of representations in the brain, the learning algorithms of the brain, and the behavioral patterns of humans and animals (Bartunov et al., 2018; Donhauser & Baillet, 2020; Michaels et al., 2019;

Original: Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018).
Revised: Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018).

Original: Evidence shows that the ability of ANNs to match real neural data depends critically on two factors.
Revised: Evidence shows that the ability of ANNs to match real neural data depends critically on two factors.

Original: First, there is a consistent correlation between the ability of an ANN to learn well on a task (e.g. image recognition, audio perception, or motor control) and the extent to which its behavior and learned representations match real data (Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018).
Revised: First, there is a consistent correlation between the ability of an ANN to learn well on a task (e.g. image recognition, audio perception, or motor control) and the extent to which its behavior and learned representations match real data (Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018).

Original: Second, the architecture of an ANN also helps to determine how well it can match real brain data, and generally, the more realistic the architecture the better the match (Schrimpf et al., 2018;
Revised: Second, the architecture of an ANN also helps to determine how well it can match real brain data, and generally, the more realistic the architecture the better the match (Schrimpf et al., 2018;

Original: Kubilius et al., 2019;
Revised: Kubilius et al., 2019;

Original: Nayebi et al., 2018).
Revised: Nayebi et al., 2018).

Original: Given these two factors, it is important for research that uses ANNs to model the brain, to use ANNs that have as realistic an architecture as possible, but which also learn well (Richards et al., 2019; Kietzmann et al., 2018; Yamins & DiCarlo, 2016).
Revised: Given these two factors, it is important for research that uses ANNs to model the brain, to use ANNs that have as realistic an architecture as possible, but which also learn well (Richards et al., 2019; Kietzmann et al., 2018; Yamins & DiCarlo, 2016).

Original: Although there are numerous disconnects between ANNs and the architecture of biological neural circuits, one of the most notable is the lack of adherence to Dale’s principle, which states that neurons release the same set of neurotransmitters from all of their synaptic terminals (Eccles, 1976).
Revised: Although there are numerous disconnects between ANNs and the architecture of biological neural circuits, one of the most notable is the lack of adherence to Dale’s principle, which states that a neuron releases the same fast neurotransmitter at all of its presynaptic terminals (Eccles, 1976).

Revised: In most circumstances, particularly in adult vertebrate brains, Dale’s principle means that neurons can only have an exclusively excitatory or inhibitory impact on their postsynaptic partners.

Original: For ANNs, this means that units cannot have a mix of positive and negative output weights, and furthermore,  weights cannot change their sign after initialisation.
Revised: For ANNs, this would mean that units cannot have a mixture of positive and negative output weights, and furthermore, that weights cannot change their sign after initialisation.

Original: In other words, units can only be excitatory or inhibitory.
Revised: In other words, units can only be excitatory or inhibitory.

Original: Some computational neuroscience studies use ANNs that have separate excitatory and inhibitory units (Song et al., 2016; Ingrosso & Abbott, 2019; Miconi, 2017; Minni et al., 2019), but many do not, particularly those that train on complicated tasks (Bartunov et al., 2018;

Original: Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf et al., 2018; Yamins et al., 2014; Kell et al.,

Original: 2018). Why do many neuroscience studies that use ANNs  not incorporate Dale’s  principle 
Revised: However, most ANNs do not incorporate Dale’s principle. Why is Dale’s principle rarely incorporated into ANNs?

Original: The reason is that this architectural constraint impairs the ability to learn—a fact that is known to many researchers who have tried to train such ANNs, but one that is rarely discussed in the literature.
Revised: The reason is that this architectural constraint impairs the ability to learn—a fact that is known to many researchers who have tried to train such ANNs, but one that is rarely discussed in the literature.

Original: However, when we seek to compare ANNs to real brains, or use them to explore biologically inspired learning rules (Bartunov et al., 2018; Whittington & Bogacz, 2019; Lillicrap et al., 2020), ideally we would have distinct populations of excitatory and inhibitory neurons, and at the same time, we would still be able to match the learning performance of standard ANNs without such constraints .
Revised: However, when we seek to compare ANNs to real brains, or use them to explore biologically inspired learning rules (Bartunov et al., 2018; Whittington & Bogacz, 2019; Lillicrap et al., 2020), ideally we would use a biologically plausible architecture with distinct populations of excitatory and inhibitory neurons, and at the same time, we would still be able to match the learning performance of standard ANNs without such constraints .

Revised: Some previous computational neuroscience studies use ANNs that have separate excitatory and inhibitory units (Song et al., 2016; Ingrosso & Abbott, 2019; Miconi, 2017; Minni et al., 2019), but these studies are focused on questions other than matching the learning performance of standard

Revised: ANNs, e.g. they focus on typical neuroscience tasks (Song et al., 2016), dynamic balance (Ingrosso & Abbott, 2019), biologically plausible learning algorithms (Miconi, 2017), or the learned structure of networks (Minni et al., 2019).

Revised: Importantly, what these papers do not do is develop means by which networks that obey Dale’s principle can match the performance of standard ANNs on machine learning benchmarks, which has become an important feature of many computational neuroscience studies using ANNs (Bartunov et al., 2018; Donhauser & Baillet, 2020; Michaels et al., 2019;

Revised: Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018).

Original: Here, we develop ANN models with separate excitatory and inhibitory units that are able to learn as well as standard ANNs.
Revised: Here, we develop ANN models with separate excitatory and inhibitory units that are able to learn as well as standard ANNs.

Original: Speciﬁcally, we develop a novel form of ANN, which we call a “Dale’s ANN” (DANN), based on feed-forward inhibition in the brain (Pouille et al., 2009).
Revised: Speciﬁcally, we develop a novel form of ANN, which we call a “Dale’s ANN” (DANN), based on feed-forward inhibition in the brain (Pouille et al., 2009).

Original: Our novel approach is different from the standard solution, which is to create ANNs with separate excitatory and inhibitory units by constraining whole columns of the weight matrix to be all positive or negative.
Revised: Our novel approach is different from the standard solution, which is to create ANNs with separate excitatory and inhibitory units by constraining whole columns of the weight matrix to be all positive or negative (Song et al., 2016).

Original: Throughout this manuscript, we refer to this standard approach as “ColumnEi” models.
Revised: Throughout this manuscript, we refer to this standard approach as “ColumnEi” models.

Original: We have departed from the ColumnEI approach in our work because it has three undesirable attributes.
Revised: We have departed from the ColumnEI approach in our work because it has three undesirable attributes.

Original: First, constrained weight matrix columns impair learning because they limit the potential solution space (Amit et al., 1989; Parisien et al., 2008).
Revised: First, constrained weight matrix columns impair learning because they limit the potential solution space (Amit et al., 1989; Parisien et al., 2008).

Original: Second, modelling excitatory and inhibitory units with the same connectivity patterns is biologically misleading, because inhibitory neurons in the brain tend to have very distinct connectivity patterns from excitatory neurons (Tremblay et al., 2016).
Revised: Second, modelling excitatory and inhibitory units with the same connectivity patterns is biologically misleading, because inhibitory neurons in the brain tend to have very distinct connectivity patterns from excitatory neurons (Tremblay et al., 2016).

Original: Third, real inhibition can act in both a subtractive and a divisive manner (Atallah et al., 2012; Wilson et al., 2012;
Revised: Third, real inhibition can act in both a subtractive and a divisive manner (Atallah et al., 2012; Wilson et al., 2012;

Original: Seybold et al., 2015; Pouille et al., 2013), which may provide important functionality.
Revised: Seybold et al., 2015; Pouille et al., 2013), which may provide important functionality.

Original: Given these considerations, in DANNs, we utilize a separate pool of inhibitory neurons with a distinct, more biologically realistic connectivity pattern, and a mixture of subtractive and divisive inhibition (Fig. 1).
Revised: Given these considerations, in DANNs, we utilize a separate pool of inhibitory neurons with a distinct, more biologically realistic connectivity pattern, and a mixture of subtractive and divisive inhibition (Fig. 1).

Original: This loosely mimics the fast feedforward subtractive and divisive inhibition provided by fast-spiking interneurons in the cortical regions of the brain (Atallah et al., 2012;
Revised: This loosely mimics the fast feedforward subtractive and divisive inhibition provided by fast-spiking interneurons in the cortical regions of the brain (Atallah et al., 2012;

Original: Hu et al., 2014;
Revised: Hu et al., 2014;

Original: Lourenço et al., 2020).
Revised: Lourenço et al., 2020).

Original: In order to get DANNs to learn as well as standard ANNs
Revised: In order to get DANNs to learn as well as standard ANNs

Original: we
Revised: we

Original: also employ two key insights:
Revised: also employ two key insights:

Original: • It is possible to view this architecture as being akin to normalisation schemes applied to the excitatory input of a layer (Ba et al., 2016; Ioffe & Szegedy, 2015; Wu & He, 2018), and we use this perspective to motivate DANN parameter initialisation.
Revised: • It is possible to view this architecture as being akin to normalisation schemes applied to the excitatory input of a layer (Ba et al., 2016; Ioffe & Szegedy, 2015; Wu & He, 2018), and we use this perspective to motivate DANN parameter initialisation.

Original: • It is important to scale the inhibitory parameter updates based on the Fisher information matrix, in order to balance the impact of excitatory and inhibitory parameter updates, similar in spirit to natural gradient approaches (Martens, 2014).
Revised: • It is important to scale the inhibitory parameter updates based on the Fisher information matrix, in order to balance the impact of excitatory and inhibitory parameter updates, similar in spirit to natural gradient approaches (Martens, 2014).

Original: Altogether, our principle contribution is a novel architecture that obey’s Dale’s principle, and that we show can learn as well as standard ANNs on machine learning benchmark tasks.
Revised: Altogether, our principle contribution is a novel architecture that obey’s Dale’s principle, and that we show can learn as well as standard ANNs on machine learning benchmark tasks.

Original: This provides the research community with a new modelling tool that will allow for more direct comparisons with real neural data than traditional ANNs allow, but which does not suffer from learning impairments.
Revised: This provides the research community with a new modelling tool that will allow for more direct comparisons with real neural data than traditional ANNs allow, but which does not suffer from learning impairments.

Original: Moreover, our results have interesting implications for inhibitory plasticity, and provide a means for future research into how excitatory and inhibitory neurons in the brain interact at the algorithmic level.
Revised: Moreover, our results have interesting implications for inhibitory plasticity, and provide a means for future research into how excitatory and inhibitory neurons in the brain interact at the algorithmic level.

Original:  B IOLOGICALLY INSPIRED NETWORKS THAT OBEY D ALE ’ S PRINCIPLE 2. M ODEL DEFINITION
Revised:  B IOLOGICALLY INSPIRED NETWORKS THAT OBEY D ALE ’ S PRINCIPLE 2. M ODEL DEFINITION

Original: Our design for DANNs takes inspiration from the physiology of feed-forward inhibitory microcircuits in the neocortex and hippocampus.
Revised: Our design for DANNs takes inspiration from the physiology of feed-forward inhibitory microcircuits in the neocortex and hippocampus.

Original: Based on these circuits, and Dale’s principle, we construct
Revised: Based on these circuits, and Dale’s principle, we construct

Original: DANNs with the following architectural constraints:
Revised: DANNs with the following architectural constraints:

Original: Each layer of the network contains two distinct populations of units, an excitatory and an inhibitory population.
Revised: Each layer of the network contains two distinct populations of units, an excitatory and an inhibitory population.

Original: There are far fewer inhibitory units than excitatory units in each layer, just as there are far more excitatory neurons than inhibitory neurons ( ∼ 5-10 times) in cortical regions of the brain 
Revised: There are far fewer inhibitory units than excitatory units in each layer, just as there are far more excitatory neurons than inhibitory neurons ( ∼ 5-10 times) in cortical regions of the brain (Tremblay et al., 2016;

Revised: Hu et al., 2014).

Original: As in real neural circuits, only the excitatory populations project between layers, and both the excitatory and inhibitory populations of layer receive excitatory projections from the layer below.
Revised: As in real neural circuits, only the excitatory populations project between layers, and both the excitatory and inhibitory populations of layer receive excitatory projections from the layer below.

Original: All of the synaptic weights are strictly non-negative, and inhibition is enforced via the activation rules for the units (eq. 1).
Revised: All of the synaptic weights are strictly non-negative, and inhibition is enforced via the activation rules for the units (eq. 1).

Original: The inhibitory population inhibits the excitatory population through a mixture of subtractive and divisive inhibition.
Revised: The inhibitory population inhibits the excitatory population through a mixture of subtractive and divisive inhibition.

Original: This constrained architecture is illustrated in Figure 1.
Revised: This constrained architecture is illustrated in Figure 1.

Original: Formally, we deﬁne the network as follows.

Original: Input to the network is received as a vector of positive scalar values x ∈ R d + , which we consider to be theﬁrst excitatory population. Each hidden layer, (cid:96) , is comprised of a vector of excitatory units  h (cid:96) ∈ R n e + and inhibitory units h
Revised: Following constraint (3), both the excitatory and inhibitory units receive inputs from the excitatory units in the layer below ( h (cid:96) − 1 ), but the inhibitory units do not project between layers.

Original: I (cid:96) ∈

Original: R n i + , in-line with constraint (1) above.

Original: Note, for the ﬁrst layer ( (cid:96) = 1 ), we have h (cid:96) = x and n e = d .

Original: Next, based on constraint (2) we set n e >> n i , and use 10% inhibitory units as default.

Original: Following constraint (3), both the excitatory and inhibitory units receive inputs from the excitatory units in the layer below ( h (cid:96) − 1 ), but the inhibitory units do not project between layers.

Original: Instead, excitatory units receive inputs from the inhibitory units of the same layer.
Revised: Instead, excitatory units receive inputs from the inhibitory units of the same layer.

Original: In-line with constraint (4), we have three sets of strictly non-negative synaptic weights, one for the excitatory connections between layers, W EE (cid:96) ∈ R n e × n e + , one for the excitatory projection to the inhibitory units W IE (cid:96) ∈ R n i × n e + , and one for the inhibitory projections within layer W EI (cid:96)
Revised: In-line with constraint (4), we have three sets of strictly non-negative synaptic weights, one for the excitatory connections between layers, W EE (cid:96) ∈ R n e × n e + , one for the excitatory projection to the inhibitory units W IE (cid:96) ∈ R n i × n e + , and one for the inhibitory projections within layer W EI (cid:96)

Original: ∈
Revised: ∈

Original: R n e × n i + .
Revised: R n e × n i + .

Original: Finally, per constraint (5), we deﬁne the impact of the inhibitory units on the excitatory units as comprising both a subtractive and a divisive component:
Revised: Finally, per constraint (5), we deﬁne the impact of the inhibitory units on the excitatory units as comprising both a subtractive and a divisive component:

Original: [Equation]
Revised: [Equation]

Original: where z E (cid:96) =
Revised: where z E (cid:96) =

Original: W EE 
Revised: W EE (cid:96) h (cid:96) −h I (cid:96) =

Original: [Equation]
Revised: f I ( z I (cid:96) ) =

Original:  h[Equation]
Revised: f I ( W IE (cid:96) h (cid:96) − 1 ) γ (cid:96) = W EI (cid:96) ( e α (cid:96) (cid:12) h

Original:  where for each layer (cid:96) , β (cid:96) ∈ R n e is a bias, g (cid:96)
Revised: I (cid:96) ) where for each layer (cid:96) , β (cid:96) ∈ R n e is a bias, g (cid:96)

Original: ∈ R n e + controls the gain, γ (cid:96) is the divisive inhibitory term, and α (cid:96) ∈ R n i is a parameter that controls the strength of this divisive inhibition.
Revised: ∈ R n e + controls the gain, γ (cid:96) is the divisive inhibitory term, and α (cid:96) ∈ R n i is a parameter that controls the strength of this divisive inhibition.

Revised: Here (cid:12) denotes elementwise multiplication (Hadamard product) and the exponential function and division are applied elementwise.

Original: In the rest of this manuscript we set f to be the rectiﬁed linear function (ReLU).
Revised: In the rest of this manuscript we set f to be the rectiﬁed linear function (ReLU).

Original: Though a ReLU function is not a perfect match to the input-output properties of real neurons, it captures the essential rectiﬁcation operation performed by neurons in physiologically realistic low activity regimes (Salinas & Sejnowski, 2000).
Revised: Though a ReLU function is not a perfect match to the input-output properties of real neurons, it captures the essential rectiﬁcation operation performed by neurons in physiologically realistic low activity regimes (Salinas & Sejnowski, 2000).

Original: In this paper, we model the inhibitory units as linear (i.e. f I ( z I ) =
Revised: In this paper, we model the inhibitory units as linear (i.e. f I ( z I ) =

Revised: z I ) since they receive only positive inputs and have no bias, and therefore their activation would always be in the linear part of the ReLU function.

Original: z I ) for mathematical simplicity,however there is some biological justiﬁcation, as the resting membrane potential of the class of fast-spiking interneurons most related to our model is relatively depolarised and their spike outputs can follow single inputs one-to-one (Hu et al., 2014; Galarreta & Hestrin, 2001).
Revised: Although we make make this modelling choice mainly for mathematical simplicity, there is some biological justiﬁcation, as the resting membrane potential of the class of fast-spiking interneurons most related to our model is relatively depolarised and their spike outputs can follow single inputs one-to-one (Hu et al., 2014; Galarreta & Hestrin, 2001).

Revised: In future work, for example in which inhibitory connections are included between inhibitory units, we expect that the use of appropriate nonlinear functions will be important.

Original:  P ARAMETER INITIALISATION FOR D ALE ’ S ANN S
Revised:  P ARAMETER INITIALISATION FOR D ALE ’ S ANN S

Original: In biology excitation and inhibition are balanced (Isaacson & Scanziani, 2011), and we use this biological property to derive appropriate weight initialisation for DANNs.
Revised: In biology excitation and inhibition are balanced (Isaacson & Scanziani, 2011), and we use this biological property to derive appropriate weight initialisation for DANNs.

Original: First we initialise excitatory parameters using W EE iid ∼ Exp
Revised: First we initialise excitatory parameters from an exponential distribution with rate parameter λ E , W EE iid ∼ Exp

Original: ( λ E ) , and then inhibitory parameters are initialised such that excitation and subtractive inhibition are balanced, i.e. E [ z E k ]
Revised: ( λ E ) , and then inhibitory parameters are initialised such that excitation and subtractive inhibition are balanced, i.e. E [ z E k ]

Original: =
Revised: =

Original: E [( W EI z I ) k ] , ∀ k .
Revised: E [( W EI z I ) k ] , ∀ k .

Original: This can be achieved in a number of ways (see appendix B.2).
Revised: This can be achieved in a number of ways (see appendix C.2).

Original: In line with biology, we choose to treat excitatory weights onto inhibitory and excitatory units the same, and sample W IE iid ∼ Exp ( λ E ) and set W EI ← 1 /n i .
Revised: In line with biology, we choose to treat excitatory weights onto inhibitory and excitatory units the same, and sample W IE iid ∼ Exp ( λ E ) and set W EI ← 1 /n i .

Original: We note that for a DANN layer with a single inhibitory neuron, e.g. at an output layer with 10 excitatory neurons, the noise inherent in sampling a single weight vector may result in a poor match between the excitatory and inhibitory inputs, so in this case we initialise W IE as 1 n e (cid:80) n e j =1 w EE j, : explicitly (where w EE j, : is the j th row of W EE ).
Revised: We note that for a DANN layer with a single inhibitory neuron, e.g. at an output layer with 10 excitatory neurons, the noise inherent in sampling a single weight vector may result in a poor match between the excitatory and inhibitory inputs, so in this case we initialise W IE as 1 n e (cid:80) n e j =1 w EE j, : explicitly (where w EE j, : is the j th row of W EE ).

Original: Next, we consider the relationship between this initialisation approach and normalisation schemes (Ba et al., 2016; Ioffe & Szegedy, 2015).
Revised: Next, we consider the relationship between this initialisation approach and normalisation schemes (Ba et al., 2016; Ioffe & Szegedy, 2015).

Original: Normalisation acts to both center and scale the unit activities in a network such that they have mean zero and variance one.
Revised: Normalisation acts to both center and scale the unit activities in a network such that they have mean zero and variance one.

Original: The weight initialisation given above will produce centered activities at the start of training.
Revised: The weight initialisation given above will produce centered activities at the start of training.

Original: We can also draw a connection between the divisive inhibition and standardisation if we assume that the elements of x are sampled from a rectiﬁed normal distribution, x iid ∼ max (0 , N (0 , σ 2 (cid:96) − 1 )) .
Revised: We can also draw a connection between the divisive inhibition and standardisation if we assume that the elements of x are sampled from a rectiﬁed normal distribution, x iid ∼ max (0 , N (0 , σ 2 (cid:96) − 1 )) .

Original: Under this assumption, the mean and variance of the excitatory input are proportional (see Appendix C).
Revised: Under this assumption, the mean and variance of the excitatory input are proportional (see Appendix D).

Original: For example, if we consider the relationship c · E [ z E k ] =
Revised: For example, if we consider the relationship c · E [ z E k ] =

Original: Var( z E k ) 1 / 2 for each unit k , we get the scalar proportionality constant c = √ 2 π − 1 / √ d , as:
Revised: Var( z E k ) 1 / 2 for each unit k , we get the scalar proportionality constant c = √ 2 π − 1 / √ d , as:

Original: [Equation]
Revised: [Equation]

Original: with expectation over the data and the parameters, and where w EE , x refer to any element of W EE , x .
Revised: with expectation over the data and the parameters, and where w EE , x refer to any element of W EE , x .

Original: Therefore, since E [ W EE ] 2 = Var( W EE ) for weights drawn from an exponential distribution, we have
Revised: Therefore, since E [ W EE ] 2 = Var( W EE ) for weights drawn from an exponential distribution, we have

Original: [Equation]
Revised: [Equation]

Original: This proportionality means that you can perform centering and standardisation operations using the same neurons.
Revised: This proportionality means that you can perform centering and standardisation operations using the same neurons.

Original: For DANNs, e α will dictate the expected variance of the layer’s activation z , as it controls the proportionality between subtractive and divisive inhibition for each inhibitory unit.
Revised: For DANNs, e α will dictate the expected variance of the layer’s activation z , as it controls the proportionality between subtractive and divisive inhibition for each inhibitory unit.

Original: If e α is set to c , then the divisive inhibition approximates dividing z E by its standard deviation, as E [ z E k ] · c = E [ w EI k, : ( e α (cid:12) z I k )]
Revised: If e α is set to c , then the divisive inhibition approximates dividing z E by its standard deviation, as E [ z E k ] · c = E [ w EI k, : ( e α (cid:12) z I k )]

Original: =
Revised: =

Original: E [ γ k ] .
Revised: E [ γ k ] .

Original: We note that due to the proportionality between z E ’s mean and variance, other values of e α will also control the layer’s variance with depth.
Revised: We note that due to the proportionality between z E ’s mean and variance, other values of e α will also control the layer’s variance with depth.

Original: However, given these considerations, we initialise e α ← √ 2 π − 1 / √ d , thereby achieving standardisation at initialisation.
Revised: However, given these considerations, we initialise e α ← √ 2 π − 1 / √ d , thereby achieving standardisation at initialisation.

Original: We ﬁnd that these initialisation schemes enable DANNs to learn well.
Revised: We ﬁnd that these initialisation schemes enable DANNs to learn well.

Original: We next turn to the question of how to perform parameter updates in DANNs in order to learn well.
Revised: We next turn to the question of how to perform parameter updates in DANNs in order to learn well.

Original:  P ARAMETER UPDATES FOR D ALE ’ S ANN S
Revised:  P ARAMETER UPDATES FOR D ALE ’ S ANN S

Original: Unlike a  column constrained network,  a layer in a DANN is not restricted in its potential function space.
Revised: Unlike a layer in a column constrained network, whose afﬁne function is restricted by sign constrained columns, a layer in a DANN is not restricted in its potential function space.

Original: This is because excitatory inputs to a layer can still have an inhibitory impact via feedforward inhibition.
Revised: This is because excitatory inputs to a layer can still have an inhibitory impact via feedforward inhibition.

Original: However, the inhibitory interneuron architecture of DANN layers introduces disparities in the degree to which updates to different parameters affect the layer’s output distribution.
Revised: However, the inhibitory interneuron architecture of DANN layers introduces disparities in the degree to which updates to different parameters affect the layer’s output distribution.

Original: This can be seen intuitively, for example if a single element of W IE is updated, this has an effect on each element of z .
Revised: This can be seen intuitively, for example if a single element of W IE is updated, this has an effect on each element of z .

Original: Similarly, an update to w EI ij will change z
Revised: Similarly, an update to w EI ij will change z

Original: i depending on the alignment of x and all of the j th inhibitory unit’s weights.
Revised: i depending on the alignment of x and all of the j th inhibitory unit’s weights.

Original: Therefore, instead of using the euclidean metric to measure distance between parameter settings, we employ an alternative approach.
Revised: Therefore, instead of using the euclidean metric to measure distance between parameter settings, we employ an alternative approach.

Original: Similar to natural gradient methods, we use an approximation of the KL divergence  of the layer’s output distribution for our metric.
Revised: Similar to natural gradient methods, we use an approximation of the Kullback-Leibler divergence (KL divergence) of the layer’s output distribution for our metric.

Original: In order to help ensure that both excitatory and inhibitory parameter updates have similar impacts on the
Revised: In order to help ensure that both excitatory and inhibitory parameter updates have similar impacts on the

Original: KL divergence, we scale the updates using correction terms derived below.
Revised: KL divergence, we scale the updates using correction terms derived below.

Original: We provide an extended derivation of these scaling factors in the Appendix D.
Revised: We provide an extended derivation of these scaling factors in the Appendix E.

Original: Given a probability distribution parameterized by some vector θ , a second order approximation to the
Revised: Given a probability distribution parameterized by some vector θ , a second order approximation to the

Original: KL divergence for a change to the parameters θ is
Revised: KL divergence for a change to the parameters θ is

Original: [Equation]
Revised: [Equation]

Original: Where F ( θ ) is the Fisher Information matrix (or just the Fisher).
Revised: Where F ( θ ) is the Fisher Information matrix (or just the Fisher).

Original: In order to calculate the Fisher for the parameters of a neural network, we must interpret the network’s outputs in a probabilistic manner.
Revised: In order to calculate the Fisher for the parameters of a neural network, we must interpret the network’s outputs in a probabilistic manner.

Original: One approach is to view a layer’s activation as parameterising a conditional distribution from the  exponential family P ( y | x ; θ ) =
Revised: One approach is to view a layer’s activation as parameterising a conditional distribution from the natural exponential family P ( y | x ; θ ) =

Original: P ( y | z ) , independent in each coordinate of y | z
Revised: P ( y | z ) , independent in each coordinate of y | z

Original: (similar to a GLM, and as done in Ba et al.
Revised: (similar to a GLM, and as done in Ba et al.

Original: The log likelihood of such a distribution can be written as1 :
Revised: The log likelihood of such a distribution can be written as

Original: [Equation]
Revised: [Equation]

Original: where f ( z ) is the activation function of the layer, and φ , η , c deﬁne the particular distribution in the exponential family.
Revised: where f ( z ) is the activation function of the layer, and φ , η , c deﬁne the particular distribution in the exponential family.

Original: Note that here we are taking η (cid:48)
Revised: Note that here we are taking η (cid:48)

Original: ( z ) and f (cid:48) ( z ) to denote ∂η∂ z and ∂f∂ z , respectively.
Revised: ( z ) and f (cid:48) ( z ) to denote ∂η∂ z and ∂f∂ z , respectively.

Original: In our networks, we have used softmax activation functions at the output and ReLU activation functions in the hidden layers.
Revised: In our networks, we have used softmax activation functions at the output and ReLU activation functions in the hidden layers.

Original: In this setting, the log likelihood of the output softmax probability layer would only be deﬁned for a one-hot vector y and would correspond to φ = 1 , c
Revised: In this setting, the log likelihood of the output softmax probability layer would only be deﬁned for a one-hot vector y and would correspond to φ = 1 , c

Original: ( y , φ ) = 0 , and η ( z )
Revised: ( y , φ ) = 0 , and η ( z )

Original: =
Revised: =

Original: log( (cid:80) i e z i ) .
Revised: log( (cid:80) i e z i ) .

Original: For the ReLU activation functions, the probabilistic model corresponds to a tobit regression model, in which y is a censored observation of a latent variable ˆ y ∼ N ( z , diag ( φf (cid:48)
Revised: For the ReLU activation functions, the probabilistic model corresponds to a tobit regression model, in which y is a censored observation of a latent variable ˆ y ∼ N ( z , diag ( φf (cid:48)

Original: ( z ))) .
Revised: ( z ))) .

Original: In this case, one could consider either the censored or pre-censored  random variable, depending on modelling preference.
Revised: In this case, one could consider either the censored or pre-censored latent random variable, depending on modelling preference.

Original: As it ﬁts well with the above framework we analyze the pre-censored random variable, i.e. f ( z ) =
Revised: As it ﬁts well with the above framework we analyze the pre-censored random variable ˆ y , i.e. f ( z ) =

Original: z in equation 6.
Revised: z in equation 6.

Original: Returning to the general case,  the Fisher of a layer is therefore:
Revised: Returning to the general case, where we consider layer’s activation as parameterising a conditional distribution from the natural exponential family, the ﬁsher of a layer is:

Original: [Equation]
Revised: [Equation]

Original: To analyse the approximate KL divergence resulting from the simple case of perturbing individual parameters of a single-layer DANN, we only need to consider the diagonal entries of the Fisher:
Revised: To analyse the approximate KL divergence resulting from the simple case of perturbing individual parameters of a single-layer DANN

Original: [Equation]
Revised: [Equation]

Revised: we only need to consider the diagonal entries of the Fisher:

Original: [Equation]
Revised: [Equation]

Original: where δ ˜ θ represents a 1 -hot vector corresponding to ˜ θ , multiplied by a scalar δ .
Revised: where δ ˜ θ represents a 1 -hot vector corresponding to ˜ θ , multiplied by a scalar δ .

Original: We now consider the approximate KL divergence after updates to a single element of W EE , W IE , W EI and α :
Revised: We now consider the approximate KL divergence after updates to a single element of W EE , W IE , W EI and α :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Where a kj = e α j γ k ( z E k − ( W EI z I ) k ) +

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: and expectations are over the data, E x ∼ P ( x ) .

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: Therefore, as a result of the feed-forward inhibitory architecture of DANNs, for a parameter update δ

Original: [Equation]
Revised: [Equation]

Original: Where a kj = e α j γ k ( z E k − ( W EI z I ) k ) + 1 , and expectations are over the data, E x ∼ P ( x ) .

Original: Therefore, as a result of the feed-forward inhibitory architecture of DANNs, for a parameter update δ , the effect on the model’s distribution will be different depending on the updated parameter-type.
Revised: the effect on the model’s distribution will be different depending on the updated parameter-type.

Original: While the exact effect depends on the degree of co-variance between terms, the most prevalent differences between and within the excitatory and inhibitory parameters are the sums over layer input and output dimensions.
Revised: While the exact effect depends on the degree of co-variance between terms, the most prevalent differences between and within the excitatory and inhibitory parameters are the sums over layer input and output dimensions.

Original: For example, an inhibitory weight update of δ to w IE ij is expected to change the model distribution approximately n e times more than an excitatory weight update of δ to w EE ij .
Revised: For example, an inhibitory weight update of δ to w IE ij is expected to change the model distribution approximately n e times more than an excitatory weight update of δ to w EE ij .

Original: In order to balance the impact of updating different parameter-types, we update DANN parameters after correcting for these terms: updates to W IE were scaled by √ n e − 1 , W EI by d − 1 and α by ( d √ n e ) − 1 .
Revised: In order to balance the impact of updating different parameter-types, we update DANN parameters after correcting for these terms: updates to W IE were scaled by √ n e − 1 , W EI by d − 1 and α by ( d √ n e ) − 1 .

Original: As a result, inhibitory unit parameters updates are scaled down relative to excitatory parameter updates.
Revised: As a result, inhibitory unit parameters updates are scaled down relative to excitatory parameter updates.

Original: This is intriguing given the differences between inhibitoryand excitatory neuron plasticity, which has been the subject of discussion in neuroscience (Kullmann & Lamsa, 2007; Kullmann et al., 2012), including the relative extent of weight changesin excitatory and inhibitory neurons (McBain et al., 1999).
Revised: This is intriguing, as while inhibitory neuron plasticity is well established, the rules and mechanisms governing synaptic updates are different from excitatory cells (Kullmann & Lamsa, 2007; Kullmann et al., 2012), and historically interneuron synapses were thought to be resistant to long-term weight changes (McBain et al., 1999).

Original: Next, we empirically veriﬁed that our heuristic correction factors captured the key differences between parameter-types in their impact on the KL divergence.
Revised: Next, we empirically veriﬁed that our heuristic correction factors captured the key differences between parameter-types in their impact on the KL divergence.

Original: To do this we compared parameter gradients before and after correction, to parameter gradients multiplied by an approximation of the diagonal of the Fisher inverse for each layer (which we refer to as Fisher corrected gradients), see Appendix E.3.
Revised: To do this we compared parameter gradients before and after correction, to parameter gradients multiplied by an approximation of the diagonal of the Fisher inverse for each layer (which we refer to as Fisher corrected gradients), see Appendix F.3.

Original: The model was trained for 50 epochs on MNIST, and updated using the Fisher corrected gradients.
Revised: The model was trained for 50 epochs on MNIST, and updated using the Fisher corrected gradients.

Original: Throughout training, we observed that the heuristic corrected gradients were more aligned to the
Revised: Throughout training, we observed that the heuristic corrected gradients were more aligned to the

Original: Fisher corrected gradients than the uncorrected gradients were (Fig. 2).
Revised: Fisher corrected gradients than the uncorrected gradients were (Fig. 2).

Original: Thus, our derived correction factors help to balance the impact of excitatory and inhibitory updates on the network’s behaviour.
Revised: Thus, our derived correction factors help to balance the impact of excitatory and inhibitory updates on the network’s behaviour.

Original: Below, we demonstrate that these corrections are key to getting DANNs to learn well.
Revised: Below, we demonstrate that these corrections are key to getting DANNs to learn well.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Having derived appropriate parameter initialisation and updates for DANNs, we now explore how they compare to traditional ANNs and ColumnEi models on simple benchmark datasets.
Revised: Having derived appropriate parameter initialisation and updates for DANNs, we now explore how they compare to traditional ANNs and ColumnEi models on simple benchmark datasets.

Original: In brief, we ﬁnd that column constrained models perform poorly, failing even to achieve zero training-set error, whereas DANNs perform equivalently to traditional ANNs.
Revised: In brief, we ﬁnd that column constrained models perform poorly, failing even to achieve zero training-set error, whereas DANNs perform equivalently to traditional ANNs.

Original: 5. I MPLEMENTATION D ETAILS
Revised: 5. I MPLEMENTATION D ETAILS

Original: All models were composed of 4 layers: in general 3 hidden layers of dimension 500 with a ReLU activation function followed by a softmax output with 10 units, and all experiments were run forepochs with batch size 32.
Revised: All models were composed of 4 layers: in general 3 hidden layers of dimension 500 with a ReLU activation function followed by a softmax output with 10 units, and all experiments were run forepochs with batch size 32.

Original: Unless stated, for DANNs and ColumnEi models, 50 inhibitory units were included per hidden layer.
Revised: Unless stated, for DANNs and ColumnEi models, 50 inhibitory units were included per hidden layer.

Original: For DANN models, the softmax output layer was constructed with one inhibitory unit.
Revised: For DANN models, the softmax output layer was constructed with one inhibitory unit.

Original: For ColumnEi models, each hidden layer’s activation is z =
Revised: For ColumnEi models, each hidden layer’s activation is z =

Original: Wx where 500 columns of W were constrained to be positive and 50 negative (therefore for ColumnEi models h (cid:96) was of dimension 550).
Revised: Wx where 500 columns of W were constrained to be positive and 50 negative (therefore for ColumnEi models h (cid:96) was of dimension 550).

Original: ColumnEi layer weights were initialised so that variance did not scale with depth and that activations were centered (see Appendix B.1 for further details).
Revised: ColumnEi layer weights were initialised so that variance did not scale with depth and that activations were centered (see Appendix C.1 for further details).

Original: All benchmark datasets (MNIST, Kuzushiji MNIST and Fashion MNIST) were pre-processed so that pixel values were in [0 , 1] .
Revised: All benchmark datasets (MNIST, Kuzushiji MNIST and Fashion MNIST) were pre-processed so that pixel values were in [0 , 1] .

Original: Learning rates were selected according to validation error averaged over 3 random seeds, after a random search (Orion; Bouthillier et al.
Revised: Learning rates were selected according to validation error averaged over 3 random seeds, after a random search (Orion; Bouthillier et al.

Original: log uniform [10, 1e-5], 100 trials, 10k validation split).
Revised: log uniform [10, 1e-5], 100 trials, 10k validation split).

Original: Selected models were then trained on test data with 6 random seeds.
Revised: Selected models were then trained on test data with 6 random seeds.

Original: Plots show mean training error per epoch, and mean test set error every 200 updates over random seeds.
Revised: Plots show mean training error per epoch, and mean test set error every 200 updates over random seeds.

Original: Tables show ﬁnal error mean ± standard deviation.
Revised: Tables show ﬁnal error mean ± standard deviation.

Original: For further implementation details see Appendix E.
Revised: For further implementation details see Appendix F.

Original: Note that because our goal in this paper is not to achieve state-of-the-art performance, we did not apply regularisation techniques, such as dropout and weight decay, or common modiﬁcations to stochastic gradient descent.
Revised: Note that because our goal in this paper is not to achieve state-of-the-art performance, we did not apply regularisation techniques, such as dropout and weight decay, or common modiﬁcations to stochastic gradient descent.

Original: Instead the goal of the experiments presented here was simply to determine whether, in the simplest test case scenario, DANNs can learn better than ColumnEi models and as well as traditional ANNs.
Revised: Instead the goal of the experiments presented here was simply to determine whether, in the simplest test case scenario, DANNs can learn better than ColumnEi models and as well as traditional ANNs.

Original: 5. C OMPARISON OF DANN S TO COLUMN - EI MODELS AND MLPS
Revised: 5. C OMPARISON OF DANN S TO COLUMN - EI MODELS AND MLPS

Original: We ﬁrst compared model performance on the MNIST dataset (Fig 3).
Revised: We ﬁrst compared model performance on the MNIST dataset (Fig 3).

Original: We observed that ColumnEi models generalised poorly, and failed to achieve 0 % training error within the 50 epochs.
Revised: We observed that ColumnEi models generalised poorly, and failed to achieve 0 % training error within the 50 epochs.

Original: This conﬁrms the fact that such models cannot learn as well as traditional ANNs.
Revised: This conﬁrms the fact that such models cannot learn as well as traditional ANNs.

Original: In contrast, we observed that DANNs performed equivalently to MLPs, and even generalised marginally better.
Revised: In contrast, we observed that DANNs performed equivalently to MLPs, and even generalised marginally better.

Revised: This was also the case for ColumnEi and DANN models constructed with 100 inhibitory units (Supp.

Original:  In addition, performance was only slightly worse when DANNswere constructed with one inhibitory unit per layer.
Revised: Fig. 6). In addition, performance was only slightly worse for DANNs with one inhibitory unit per layer.

Revised: These results show that DANN performance generalizes to different ratios of excitatory-to-inhibitory units.

Original: We also found that not correcting parameter updates using the corrections derived from the Fisher signiﬁcantly impaired optimization, further verifying the correction factors (see also Fig 2).
Revised: We also found that not correcting parameter updates using the corrections derived from the Fisher signiﬁcantly impaired optimization, further verifying the correction factors (see also Fig 2).

Original: Next, we compared DANN performance to MLPs trained with batch and layer normalization on more challenging benchmark datasets (Fig 4).
Revised: Next, we compared DANN performance to MLPs trained with batch and layer normalization on more challenging benchmark datasets (Fig 4).

Original: Again we found that DANNs performed equivalently to these standard architectures, whereas ColumnEi models struggled to achieve acceptable performance.
Revised: Again we found that DANNs performed equivalently to these standard architectures, whereas ColumnEi models struggled to achieve acceptable performance.

Original: Finally, we explored methods for improving DANN performance.
Revised: Finally, we explored methods for improving DANN performance.

Original: First, in order to maintain the positive DANN weight constraint, if after a parameter update a weight was negative, we reset it to zero, i.e. θ ← max(0 , θ ) , and as a result the actual update is no longer that suggested by stochastic gradient descent.
Revised: First, in order to maintain the positive DANN weight constraint, if after a parameter update a weight was negative, we reset it to zero, i.e. θ ← max(0 , θ ) , and as a result the actual update is no longer that suggested by stochastic gradient descent.

Original: We therefore experimented with temporarily reducing the learning rate whenever this parameter clipping would reduce the cosine of the angle made between the gradient and actual updates below a certain constraint (see Appendix E.4).
Revised: We therefore experimented with temporarily reducing the learning rate whenever this parameter clipping would reduce the cosine of the angle made between the gradient and actual updates below a certain constraint (see Appendix F.4).

Original: Second, we note that the divisive inhibition term, γ , appears in the denominator of the weight gradients (Appendix D.2) and, therefore, if γ becomes small, the gradients will become large, potentially resulting in inappropriate parameter updates.
Revised: Second, we note that the divisive inhibition term, γ , appears in the denominator of the weight gradients (Appendix E.2) and, therefore, if γ becomes small, the gradients will become large, potentially resulting in inappropriate parameter updates.

Original: We therefore wondered if gradient normalisation would be particularly effective for DANNs.
Revised: We therefore wondered if gradient normalisation would be particularly effective for DANNs.

Original: We tested both of these modiﬁcations to DANNs trained on FashionMNIST (Supp.
Revised: We tested both of these modiﬁcations to DANNs trained on FashionMNIST (Supp.

Original: Fig. 5).
Revised: Fig. 5).

Original: However, we found that they provided no observable improvement, indicating that the loss landscape and gradients were well behaved over optimization.
Revised: However, we found that they provided no observable improvement, indicating that the loss landscape and gradients were well behaved over optimization.

Original: Altogether, our results demonstrate that: (1) the obvious approach to creating ANNs that obey Dale’s principle (ColumnEi models) do not learn as well as traditional ANNs, (2) DANNs learn better than
Revised: Altogether, our results demonstrate that: (1) the obvious approach to creating ANNs that obey Dale’s principle (ColumnEi models) do not learn as well as traditional ANNs, (2) DANNs learn better than

Original: ColumnEi models and as well as traditional ANNs, (3) DANN learning is signiﬁcantly improved by taking appropriate steps to scale updates in excitatory and inhibitory units appropriately.
Revised: ColumnEi models and as well as traditional ANNs, (3) DANN learning is signiﬁcantly improved by taking appropriate steps to scale updates in excitatory and inhibitory units appropriately.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Here we presented DANNs, a novel ANN architecture with separate inhibitory and excitatory units.
Revised: Here we presented DANNs, a novel ANN architecture with separate inhibitory and excitatory units.

Original: We derived appropriate parameter initialisation and update rules and showed experimentally that, unlike ANNs where some columns are simply constrained to be positive or negative, DANNs perform equivalently to traditional ANNs on benchmark datasets.
Revised: We derived appropriate parameter initialisation and update rules and showed experimentally that, unlike ANNs where some columns are simply constrained to be positive or negative, DANNs perform equivalently to traditional ANNs on benchmark datasets.

Original: These results are important as they are, as far as we know, the ﬁrst example of an ANN architecture that fully adheres to Dale’s law without sacriﬁcing learning performance .
Revised: These results are important as they are, as far as we know, the ﬁrst example of an ANN architecture that fully adheres to Dale’s law without sacriﬁcing learning performance .

Original:  There are a number of avenues for future work building upon DANNs, the most obvious of which are to extend and incorporate DANNs into recurrent and convolution neural networks.
Revised: However, our results also raise an interesting question: why does nature employ Dale’s principle? After all, we did not see any improvement over normal ANNs in our experiments. There are two possible hypotheses. First, it is possible that Dale’s principle represents an evolutionary local minima, whereby early phylogenetic choices led to constraints on the system that were difﬁcult to escape via natural selection.

Original: An alternative direction is explore more fully the balance of excitation and  inhibition during optimization, as while
Revised: Alternatively, Dale’s principle may provide some computational beneﬁt that we were unable to uncover given the speciﬁc tasks and architectures we used here. For example, it has been hypothesized that inhibition may help to prevent catastrophic forgetting (Barron et al., 2017). We consider exploring these questions an important avenue for future research. There are a number of additional avenues for future work building upon DANNs, the most obvious of which are to extend and incorporate DANNs into recurrent and convolution neural networks (see

Original:  DANNs are initialised such that these are balanced,  the inhibitory parameters are updated during training, and the model is free to diverge from this initialisation.
Revised: Appendix B for a discussion of these extensions, as well as preliminary results). It would also be interesting to explore the relative roles of subtractive and divisive inhibition. While subtractive inhibition is required for the unconstrained functional space of DANN layers, divisive inhibition may confer some of the same optimisation beneﬁts as normalisation schemes. A related issue would be to explore the continued balance of excitation and inhibition during optimization, as while DANNs are initialised such that these are balanced, and inhibition approximates normalisation schemes, the inhibitory parameters are updated during training, and the model is free to diverge from this initialisation.

Original: As a result, the distribution of layer activations may be unstable over successive parameter updates, potentially harming optimization.
Revised: As a result, the distribution of layer activations may be unstable over successive parameter updates, potentially harming optimization.

Original: In the brain, a variety of homeostatic plasticity mechanisms stabilize neuronal activity.
Revised: In the brain, a variety of homeostatic plasticity mechanisms stabilize neuronal activity.

Original: For example, reducing excitatory input naturally results in a reduction in inhibition in real neural circuits (Tien & Kerschensteiner, 2018).
Revised: For example, reducing excitatory input naturally results in a reduction in inhibition in real neural circuits (Tien & Kerschensteiner, 2018).

Original: It would therefore be interesting to test the inclusion of a homeostatic loss to encourage inhibition to track excitation throughout training.
Revised: It would therefore be interesting to test the inclusion of a homeostatic loss to encourage inhibition to track excitation throughout training.

Revised: Finally, we note that while fast feedforward inhibition in the mammalian cortex was the main source of inspiration for this work, future investigations may beneﬁt from drawing on a broader range of neurobiology, for example by incorporating principles of invertebrate neural circuits, such as the mushroom bodies of insects (Serrano et al., 2013).

Original: In summary, DANNs sit at the intersection of a number of programs of research.
Revised: In summary, DANNs sit at the intersection of a number of programs of research.

Original: First, they are a new architecture that obeys Dale’s principle, but which can still learn well, allowing researchers to more directly compare trained ANNs to real neural data (Schrimpf et al., 2018; Yamins et al., 2014).
Revised: First, they are a new architecture that obeys Dale’s principle, but which can still learn well, allowing researchers to more directly compare trained ANNs to real neural data (Schrimpf et al., 2018; Yamins et al., 2014).

Original: Second,
Revised: Second,

Original: DANNs contribute towards computational neuroscience and machine learning work on inhibitory interneurons in ANNs, and in general towards the role of inhibitory circuits and plasticity in neural computation (Song et al., 2016;
Revised: DANNs contribute towards computational neuroscience and machine learning work on inhibitory interneurons in ANNs, and in general towards the role of inhibitory circuits and plasticity in neural computation (Song et al., 2016;

Original: Sacramento et al., 2018; Costa et al., 2017; Payeur et al., 2020;
Revised: Sacramento et al., 2018; Costa et al., 2017; Payeur et al., 2020;

Original: Atallah et al., 2012; Barron et al., 2017).
Revised: Atallah et al., 2012; Barron et al., 2017).

Original: Finally, the inhibition in DANNs also has an interesting connection to normalisation methods used to improving learning in deep networks (Ioffe & Szegedy, 2015; Wu & He, 2018; Ba et al., 2016).
Revised: Finally, the inhibition in DANNs also has an interesting connection to normalisation methods used to improving learning in deep networks (Ioffe & Szegedy, 2015; Wu & He, 2018; Ba et al., 2016).

Original: As DANNs tie these distinct programs of research together into a single model, we hope they can serve as a basis for future research at the intersection of deep learning and neuroscience.
Revised: As DANNs tie these distinct programs of research together into a single model, we hope they can serve as a basis for future research at the intersection of deep learning and neuroscience.