Original: S PARSE D ISTRIBUTED M EMORY IS 
Revised: S PARSE D ISTRIBUTED M EMORY IS A C ONTINUAL L EARNER

Original: A
Revised: alexander

Original: C ONTINUAL L EARNER Anonymous authors Paper under double-blind review
Revised: Dmitry Krotov

Original: A BSTRACT
Revised: A BSTRACT

Original: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving.
Revised: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving.

Original: Building on work using Sparse Distributed
Revised: Building on work using Sparse Distributed

Original: Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner.
Revised: Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner.

Original: We find that every component of our MLP variant translated from biology is necessary for continual learning.
Revised: We find that every component of our MLP variant translated from biology is necessary for continual learning.

Original: Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.
Revised: Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Biological networks tend to thrive in continually learning novel tasks, a problem that remains daunting for artificial neural networks.
Revised: Biological networks tend to thrive in continually learning novel tasks, a problem that remains daunting for artificial neural networks.

Original: Here, we use Sparse Distributed Memory (SDM) to modify a
Revised: Here, we use Sparse Distributed Memory (SDM) to modify a

Original: Multi-Layered Perceptron (MLP) with features from a cerebellum-like neural circuit that are shared across organisms as diverse as humans, fruit flies, and electric fish (Modi et al., 2020; Xie et al., 2022).
Revised: Multi-Layered Perceptron (MLP) with features from a cerebellum-like neural circuit that are shared across organisms as diverse as humans, fruit flies, and electric fish (Modi et al., 2020; Xie et al., 2022).

Original: These modifications result in a new MLP variant (referred to as SDMLP) that uses a TopK 1 activation function (keeping only the k most excited neurons in a layer on), no bias terms, and enforces both L 2 normalization and non-negativity constraints on its weights and data.
Revised: These modifications result in a new MLP variant (referred to as SDMLP) that uses a TopK 1 activation function (keeping only the k most excited neurons in a layer on), no bias terms, and enforces both L 2 normalization and non-negativity constraints on its weights and data.

Original: All of these
Revised: All of these

Original: SDM-derived components are necessary for our model to avoid catastrophic forgetting.
Revised: SDM-derived components are necessary for our model to avoid catastrophic forgetting.

Original: We encounter challenges when training the SDMLP that we leverage additional neurobiology to solve, resulting in better continual learning performance.
Revised: We encounter challenges when training the SDMLP that we leverage additional neurobiology to solve, resulting in better continual learning performance.

Original: Our first problem is with “dead neurons” that are never active for any input and which are caused by the Top-K activation function (Makhzani & Frey, 2014; Ahmad & Scheinkman, 2019).
Revised: Our first problem is with “dead neurons” that are never active for any input and which are caused by the Top-K activation function (Makhzani & Frey, 2014; Ahmad & Scheinkman, 2019).

Original: Having fewer neurons participating in learning results in more of them being overwritten by any new continual learning task, increasing catastrophic forgetting.
Revised: Having fewer neurons participating in learning results in more of them being overwritten by any new continual learning task, increasing catastrophic forgetting.

Original: Our solution imitates the “GABA Switch” phenomenon where inhibitory interneurons that implement Top-K will excite rather than inhibit early in development (Gozel & Gerstner, 2021).
Revised: Our solution imitates the “GABA Switch” phenomenon where inhibitory interneurons that implement Top-K will excite rather than inhibit early in development (Gozel & Gerstner, 2021).

Original: The second problem is with optimizers that use momentum, which becomes “stale” when training highly sparse networks.
Revised: The second problem is with optimizers that use momentum, which becomes “stale” when training highly sparse networks.

Original: This staleness refers to the optimizer continuing to update inactive neurons with an out-of-date moving average, killing neurons and again harming continual learning.
Revised: This staleness refers to the optimizer continuing to update inactive neurons with an out-of-date moving average, killing neurons and again harming continual learning.

Original: To our knowledge, we are the first to formally identify this problem that will in theory affect any sparse model, including recent Mixtures of Experts (Fedus et al., 2021;
Revised: To our knowledge, we are the first to formally identify this problem that will in theory affect any sparse model, including recent Mixtures of Experts (Fedus et al., 2021;

Original: Shazeer et al., 2017; Roller et al., 2021).
Revised: Shazeer et al., 2017; Roller et al., 2021).

Original: Our SDMLP is a strong continual learner, especially when combined with complementary approaches.
Revised: Our SDMLP is a strong continual learner, especially when combined with complementary approaches.

Original: Using our solution in conjunction with Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), we obtain, to the best of our knowledge, state-of-the-art performance for CIFARin the class incremental setting when memory replay is not allowed.
Revised: Using our solution in conjunction with Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), we obtain, to the best of our knowledge, state-of-the-art performance for CIFARin the class incremental setting when memory replay is not allowed.

Original: Another variant of SDM, developed independently of our work, appears to be state-of-the-art for CIFAR-100, MNIST, and
Revised: Another variant of SDM, developed independently of our work, appears to be state-of-the-art for CIFAR-100, MNIST, and

Original: FashionMNIST, with our SDMLP as a close second (Shen et al., 2021).
Revised: FashionMNIST, with our SDMLP as a close second (Shen et al., 2021).

Original: Excitingly, our continual learning success is “organic” in resulting from the underlying model architecture and does not require any task labels, task boundaries, or memory replay.
Revised: Excitingly, our continual learning success is “organic” in resulting from the underlying model architecture and does not require any task labels, task boundaries, or memory replay.

Original: Abstractly, SDM learns its subnetworks responsible for continual learning using two core model components.
Revised: Abstractly, SDM learns its subnetworks responsible for continual learning using two core model components.

Original: First, the Top-K activation function causes the k neurons most activated by an input to specialize towards this input, resulting in the formation of specialized subnetworks.
Revised: First, the Top-K activation function causes the k neurons most activated by an input to specialize towards this input, resulting in the formation of specialized subnetworks.

Original: Second, the L 2 normalization and absence of a bias term together constrain neurons to the data manifold, ensuring that all neurons democratically participate in learning.
Revised: Second, the L 2 normalization and absence of a bias term together constrain neurons to the data manifold, ensuring that all neurons democratically participate in learning.

Original: When these two components are combined, a new learning task only activates and trains a small subset of neurons, leaving the rest of the network intact to remember previous tasks without being overwritten
Revised: When these two components are combined, a new learning task only activates and trains a small subset of neurons, leaving the rest of the network intact to remember previous tasks without being overwritten

Original: As a roadmap of the paper, we first discuss related work (Section 2).
Revised: As a roadmap of the paper, we first discuss related work (Section 2).

Original: We then provide a short introduction to SDM (Section 3), before translating it into our MLP (Section 4).
Revised: We then provide a short introduction to SDM (Section 3), before translating it into our MLP (Section 4).

Original: Next we present our results, comparing the organic continual learning capabilities of our SDMLP against relevant benchmarks (Section 5).
Revised: Next we present our results, comparing the organic continual learning capabilities of our SDMLP against relevant benchmarks (Section 5).

Original: Finally, we conclude with a discussion on the limitations of our work, sparse models more broadly, and how SDM relates MLPs to Transformer Attention (Section 6).
Revised: Finally, we conclude with a discussion on the limitations of our work, sparse models more broadly, and how SDM relates MLPs to Transformer Attention (Section 6).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Continual Learning - The techniques developed for continual learning can be broadly divided into three categories: architectural (Goodfellow et al., 2014; 2013), regularization (Smith et al., 2022;
Revised: Continual Learning - The techniques developed for continual learning can be broadly divided into three categories: architectural (Goodfellow et al., 2014; 2013), regularization (Smith et al., 2022;

Original: Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018), and rehearsal (Lange et al., 2021;
Revised: Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018), and rehearsal (Lange et al., 2021;

Original: Hsu et al., 2018).
Revised: Hsu et al., 2018).

Original: Many of these approaches have used the formation of sparse subnetworks for continual learning (Abbasi et al., 2022; Aljundi et al., 2019b; Ramasesh et al., 2022; Iyer et al., 2022; Xu & Zhu, 2018; Mallya & Lazebnik, 2018; Schwarz et al., 2021; Smith et al., 2022).
Revised: Many of these approaches have used the formation of sparse subnetworks for continual learning (Abbasi et al., 2022; Aljundi et al., 2019b; Ramasesh et al., 2022; Iyer et al., 2022; Xu & Zhu, 2018; Mallya & Lazebnik, 2018; Schwarz et al., 2021; Smith et al., 2022).

Original: However, in contrast to our “organic” approach, these methods employ complex algorithms and additional memory consumption to explicitly protect model weights important for previous tasks from overwrites.
Revised: However, in contrast to our “organic” approach, these methods employ complex algorithms and additional memory consumption to explicitly protect model weights important for previous tasks from overwrites.

Original: Works applying the Top-K activation to continual learning include (Aljundi et al., 2019b; Gozel &
Revised: Works applying the Top-K activation to continual learning include (Aljundi et al., 2019b; Gozel &

Original: Gerstner, 2021; Iyer et al., 2022).
Revised: Gerstner, 2021; Iyer et al., 2022).

Original: Srivastava et al.
Revised: Srivastava et al.

Original: (2013) used a local version of Top-K, defining disjoint subsets of neurons in each layer and applying Top-K locally to each.
Revised: (2013) used a local version of Top-K, defining disjoint subsets of neurons in each layer and applying Top-K locally to each.

Original: However, this was only used on a simple task-incremental two-split MNIST task and without any of the additional
Revised: However, this was only used on a simple task-incremental two-split MNIST task and without any of the additional

Original: SDM modifications that we found crucial to our strong performance (Table 2).
Revised: SDM modifications that we found crucial to our strong performance (Table 2).

Original: The Top-K activation function has also been applied more broadly in deep learning (Makhzani &
Revised: The Top-K activation function has also been applied more broadly in deep learning (Makhzani &

Original: Frey, 2014; Ahmad & Scheinkman, 2019; Sengupta et al., 2018; Gozel & Gerstner, 2021; Aljundi et al., 2019b).
Revised: Frey, 2014; Ahmad & Scheinkman, 2019; Sengupta et al., 2018; Gozel & Gerstner, 2021; Aljundi et al., 2019b).

Original: Top-K converges not only with the connectivity of many brain regions that utilize inhibitory interneurons but also with existing results showing that Top-K has a number of advantages beyond continual learning, including: greater interpretability (Makhzani & Frey, 2014; Krotov &
Revised: Top-K converges not only with the connectivity of many brain regions that utilize inhibitory interneurons but also with existing results showing that Top-K has a number of advantages beyond continual learning, including: greater interpretability (Makhzani & Frey, 2014; Krotov &

Original: Hopfield, 2019; Grinberg et al., 2019), robustness to adversarial attacks (Paiton et al., 2020; Krotov & Hopfield, 2018; Iyer et al., 2022), efficient sparse computations (Ahmad & Scheinkman, 2019), tiling of the data manifold (Sengupta et al., 2018), and implementation with local Hebbian learning rules (Gozel & Gerstner, 2021; Sengupta et al., 2018; Krotov & Hopfield, 2019; Ryali et al., 2020;
Revised: Hopfield, 2019; Grinberg et al., 2019), robustness to adversarial attacks (Paiton et al., 2020; Krotov & Hopfield, 2018; Iyer et al., 2022), efficient sparse computations (Ahmad & Scheinkman, 2019), tiling of the data manifold (Sengupta et al., 2018), and implementation with local Hebbian learning rules (Gozel & Gerstner, 2021; Sengupta et al., 2018; Krotov & Hopfield, 2019; Ryali et al., 2020;

Original: Liang et al., 2020).
Revised: Liang et al., 2020).

Original: FlyModel - The most closely related method to ours is a model of the Drosophila Mushroom Body circuitry (Shen et al., 2021).
Revised: FlyModel - The most closely related method to ours is a model of the Drosophila Mushroom Body circuitry (Shen et al., 2021).

Original: This model, referred to as “FlyModel”, unknowingly implements  the SDM algorithm, specifically the Hyperplane variant (Jaeckel, 1989a) with a Top-K activation function that we also use and will justify (Keeler, 1988).
Revised: This model, referred to as “FlyModel”, unknowingly implements Memory replay methods will indirectly determine and protect model weights through deciding what mem- ories to replay. the SDM algorithm, specifically the Hyperplane variant (Jaeckel, 1989a) with a Top-K activation function that we also use and will justify (Keeler, 1988).

Original: The FlyModel shows strong continual learning performance, trading off the position of best performer with our SDMLP across tasks and bolstering the title of our paper that SDM is a continual learner.
Revised: The FlyModel shows strong continual learning performance, trading off the position of best performer with our SDMLP across tasks and bolstering the title of our paper that SDM is a continual learner.

Original: While both models are derived from SDM, our work both extends the theory of SDM by allowing it to successfully learn data manifolds (Appendix A.4), and reconciles the differences between SDM and MLPs, such that SDM can be trained in the deep learning framework (this includes having no fixed neuron weights and using backpropagation).
Revised: While both models are derived from SDM, our work both extends the theory of SDM by allowing it to successfully learn data manifolds (App. A.4) , and reconciles the differences between SDM and MLPs, such that SDM can be trained in the deep learning framework (this includes having no fixed neuron weights and using backpropagation).

Original: Both of these contributions are novel and may inspire future work beyondthe scope of continual learning.
Revised: Both of these contributions are novel and may inspire future work beyond continual learning.

Original: For example, learning the data manifold preserves similarity in the data and leads to more specialized, interpretable neurons (e.g. Figureof Appendix H).
Revised: For example, learning the data manifold preserves similarity in the data and leads to more specialized, interpretable neurons (e.g. Fig. 26 of App. H).

Original: Training in the deep learning framework also allows us to combine SDMLP with other gradientbased methods like Elastic Weight Consolidation (EWC) that the FlyModel is incompatible with (Kirkpatrick et al., 2017).
Revised: Training in the deep learning framework also allows us to combine SDMLP with other gradientbased methods like Elastic Weight Consolidation (EWC) that the FlyModel is incompatible with (Kirkpatrick et al., 2017).

Original: Additionally, demonstrating how MLPs can be implemented as a cerebellar circuit serves as an example for how ideas from neuroscience, like the GABA switch, can be leveraged to the benefit of deep learning.
Revised: Additionally, demonstrating how MLPs can be implemented as a cerebellar circuit serves as an example for how ideas from neuroscience, like the GABA switch, can be leveraged to the benefit of deep learning.

Original:  B ACKGROUND ON S PARSE D ISTRIBUTED M EMORY
Revised:  B ACKGROUND ON S PARSE D ISTRIBUTED M EMORY

Original: Sparse Distributed Memory (SDM) is an associative memory model that tries to solve the problem of how patterns (memories) could be stored in the brain (Kanerva, 1988; 1993) and has with close connections to Hopfield networks, the circuitry of the cerebellum, and Transformer Attention (Kanerva, 1988; Bricken & Pehlevan, 2021; Hopfield, 1982; 1984; Krotov & Hopfield, 2016;
Revised: Sparse Distributed Memory (SDM) is an associative memory model that tries to solve the problem of how patterns (memories) could be stored in the brain (Kanerva, 1988; 1993) and has with close connections to Hopfield networks, the circuitry of the cerebellum, and Transformer Attention (Kanerva, 1988; Bricken & Pehlevan, 2021; Hopfield, 1982; 1984; Krotov & Hopfield, 2016;

Original: Tyulmankov et al., 2021; Millidge et al., 2022).
Revised: Tyulmankov et al., 2021; Millidge et al., 2022).

Original: We briefly provide background on SDM and notation sufficient to relate SDM to MLPs.
Revised: We briefly provide background on SDM and notation sufficient to relate SDM to MLPs.

Original: For a summary of how SDM relates to the cerebellum, see Appendix A.2.
Revised: For a summary of how SDM relates to the cerebellum, see App. A.2.

Original: We use the continuous version of SDM, where all neurons and patterns exist on the L 2 unit norm hypersphere and cosine similarity is our distance metric
Revised: We use the continuous version of SDM, where all neurons and patterns exist on the L 2 unit norm hypersphere and cosine similarity is our distance metric

Original: (Bricken & Pehlevan, 2021).
Revised: (Bricken & Pehlevan, 2021).

Original: SDM randomly initializes the addresses of r neurons on the L 2 unit hypersphere in an n dimensional space.
Revised: SDM randomly initializes the addresses of r neurons on the L 2 unit hypersphere in an n dimensional space.

Original: These neurons have addresses that each occupy a column in our address matrix
Revised: These neurons have addresses that each occupy a column in our address matrix

Original: X a ∈ ( L 2 ) n × r , where L 2 is shorthand for all n -dimensional vectors existing on the L 2 unit norm hypersphere.
Revised: X a ∈ ( L 2 ) n × r , where L 2 is shorthand for all n -dimensional vectors existing on the L 2 unit norm hypersphere.

Original: Each neuron also has a storage vector used to store patterns represented in the matrix
Revised: Each neuron also has a storage vector used to store patterns represented in the matrix

Original: X v ∈ R o × r , where o is the output dimension.
Revised: X v ∈ R o × r , where o is the output dimension.

Original: Patterns also have addresses constrained on the ndimensional L 2 hypersphere determined by their encoding; encodings can be as simple as flattening an image into a vector or as complex as preprocessing with a deep learning model.
Revised: Patterns also have addresses constrained on the ndimensional L 2 hypersphere determined by their encoding; encodings can be as simple as flattening an image into a vector or as complex as preprocessing with a deep learning model.

Original: Patterns are stored by activating all nearby neurons within a cosine similarity threshold c , and performing an elementwise summation with the activated neurons’ storage vector.
Revised: Patterns are stored by activating all nearby neurons within a cosine similarity threshold c , and performing an elementwise summation with the activated neurons’ storage vector.

Original: Depending on the task at hand, patterns write themselves into the storage vector (e.g., during a reconstruction task) or write another pattern, possibly of different dimension (e.g., writing in their one hot label for a classification task).
Revised: Depending on the task at hand, patterns write themselves into the storage vector (e.g., during a reconstruction task) or write another pattern, possibly of different dimension (e.g., writing in their one hot label for a classification task).

Original: This write operation and the ensuing read operation are summarized in Fig.
Revised: This write operation and the ensuing read operation are summarized in Fig.

Original: Because in most cases we have fewer neurons than patterns, the same neuron will be activated by multiple different patterns.
Revised: Because in most cases we have fewer neurons than patterns, the same neuron will be activated by multiple different patterns.

Original: This is handled by storing the pattern values in superposition via the aforementioned elementwise summation operation.
Revised: This is handled by storing the pattern values in superposition via the aforementioned elementwise summation operation.

Original: The fidelity of each pattern stored in this superposition is a function of the vector orthogonality and dimensionality n .
Revised: The fidelity of each pattern stored in this superposition is a function of the vector orthogonality and dimensionality n .

Original: Using m to denote the number of patterns, matrix P a ∈ ( L 2 ) n × m for the pattern addresses, and matrix P v ∈ R o × m for the values patterns want to write, the SDM write operation is:
Revised: Using m to denote the number of patterns, matrix P a ∈ ( L 2 ) n × m for the pattern addresses, and matrix P v ∈ R o × m for the values patterns want to write, the SDM write operation is:

Original: [Equation]
Revised: [Equation]

Original: where b ( e ) performs an element-wise binarization of its input to determine which pattern and neuron addresses are within the cosine similarity threshold c of each other.
Revised: where b ( e ) performs an element-wise binarization of its input to determine which pattern and neuron addresses are within the cosine similarity threshold c of each other.

Original: Having written patterns into our neurons, we read from the system by inputting a query ξ , that again activates nearby neurons.
Revised: Having written patterns into our neurons, we read from the system by inputting a query ξ , that again activates nearby neurons.

Original: Each activated neuron outputs its storage vector and they are all summed elementwise to give a final output y .
Revised: Each activated neuron outputs its storage vector and they are all summed elementwise to give a final output y .

Original: The output y can be interpreted as an updated query and optionally L 2 normalized again as a post processing step:
Revised: The output y can be interpreted as an updated query and optionally L 2 normalized again as a post processing step:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, SDM’s query will update towards the values of the patterns with the closest addresses.
Revised: Intuitively, SDM’s query will update towards the values of the patterns with the closest addresses.

Original: This is because the patterns with the closest addresses will have written their values into more neurons that the query reads from than any competing patterns.
Revised: This is because the patterns with the closest addresses will have written their values into more neurons that the query reads from than any competing patterns.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 1, the blue pattern address is the closest to the query meaning that it appears the most in those nearby neurons the query reads from.
Revised: 1, the blue pattern address is the closest to the query meaning that it appears the most in those nearby neurons the query reads from.

Original: SDM is closely related to modern Hopfield networks (Krotov & Hopfield, 2016; Ramsauer et al., 2020; Krotov & Hopfield, 2020) if they are restricted to a single step update of the recurrent dynamics (Tyulmankov et al., 2021;
Revised: SDM is closely related to modern Hopfield networks (Krotov & Hopfield, 2016; Ramsauer et al., 2020; Krotov & Hopfield, 2020) if they are restricted to a single step update of the recurrent dynamics (Tyulmankov et al., 2021;

Original: Bricken & Pehlevan, 2021; Millidge et al., 2022).
Revised: Bricken & Pehlevan, 2021; Millidge et al., 2022).

Original:  T RANSLATING SDM INTO MLP S FOR CONTINUAL LEARNING
Revised:  T RANSLATING SDM INTO MLP S FOR CONTINUAL LEARNING

Original: A one hidden layer MLP transforms an input ξ to an output y .
Revised: A one hidden layer MLP transforms an input ξ to an output y .

Original: Using notation compatible with SDM and representing unnormalized continuous vectors with a tilde, we can write the MLP as:
Revised: Using notation compatible with SDM and representing unnormalized continuous vectors with a tilde, we can write the MLP as:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X a ∈ R n × r and ˜ X v ∈ R o × r are weight matrices corresponding to our SDM neuron addresses and values, respectively.
Revised: where ˜ X a ∈ R n × r and ˜ X v ∈ R o × r are weight matrices corresponding to our SDM neuron addresses and values, respectively.

Original: Meanwhile, ˜ b a ∈ R r , ˜ b v ∈ R o are the bias parameters and f ( · ) is the activation function used by the MLP such as ReLU (Glorot et al., 2011).
Revised: Meanwhile, ˜ b a ∈ R r , ˜ b v ∈ R o are the bias parameters and f ( · ) is the activation function used by the MLP such as ReLU (Glorot et al., 2011).

Original: Using this SDM notation for the MLP, it is trivial to see the similarity between the SDM read Eq. 2 and Eq.
Revised: Using this SDM notation for the MLP, it is trivial to see the similarity between the SDM read Eq.and Eq.

Original: 3 for single hidden layer MLPs that was first established in (Kanerva, 1993).
Revised: 3 for single hidden layer MLPs that was first established in (Kanerva, 1993).

Original: However, the fixed random neuron address of SDM makes it unable to effectively model real-world data.
Revised: However, the fixed random neuron address of SDM makes it unable to effectively model real-world data.

Original: Using ideas from Keeler (1988), we resolve this inability with a biologically plausible inhibitory interneuron that is approximated by a Top-K activation function.
Revised: Using ideas from Keeler (1988), we resolve this inability with a biologically plausible inhibitory interneuron that is approximated by a Top-K activation function.

Original: Appendix A.3 explains how SDM is modified and the actual Top-K activation function used is presented shortly in Eq.
Revised: App. A.3 explains how SDM is modified and the actual Top-K activation function used is presented shortly in Eq.

Original: This modification makes SDM compatible with an MLP that has no fixed weights.
Revised: This modification makes SDM compatible with an MLP that has no fixed weights.

Original: As for the SDM write operation of Eq.
Revised: As for the SDM write operation of Eq.

Original: 1, this is related to an MLP trained with backprop as outlined in Appendix A.6.
Revised: 1, this is related to an MLP trained with backprop as outlined in App. A.6.

Original: The Dead Neuron Problem - An issue with the Top-K activation function is that it creates dead neurons (Ahmad & Scheinkman, 2019;Abbasi et al., 2022; van den Oord et al., 2017; Sengupta et al., 2018; Rumelhart & Zipser, 1985; Makhzani & Frey, 2014; Fedus et al., 2021).
Revised: The Dead Neuron Problem - An issue with the Top-K activation function is that it creates dead neurons (Ahmad & Scheinkman, 2019; Rumelhart & Zipser, 1985; Makhzani & Frey, 2014; Fedus et al., 2021).

Original: Only a subset of the randomly initialized neurons will exist closest to the data manifold andalways be in the top k most active, leaving all remaining neurons to never be activated or update their weights.
Revised: Only a subset of the randomly initialized neurons will exist closest to the data manifold and be in the top k most active, leaving all remaining neurons to never be activated.

Original: In the continual learning setting, when a new task is introduced,this means the model has fewer neurons that can learn and must overwrite moreof those that were used for the previous task(s), resulting in catastrophic forgetting.
Revised: In the continual learning setting, when a new task is introduced, the model has fewer neurons that can learn and must overwrite more that were used for the previous task(s), resulting in catastrophic forgetting.

Original: Rather than pre-initializing our neuron weights using a decomposition of the data manifold (Rumelhart & Zipser, 1985; McInnes & Healy, 2018; Strang, 1993)
Revised: Rather than pre-initializing our neuron weights using a decomposition of the data manifold (Rumelhart & Zipser, 1985; McInnes & Healy, 2018; Strang, 1993)

Original: we ensure that all neurons are active at the start of training so that they update onto the manifold.
Revised: we ensure that all neurons are active at the start of training so that they update onto the manifold.

Original: This approach has been used before but in biologically implausible ways (see Appendix B.1)
Revised: This approach has been used before but in biologically implausible ways (see App. B.1)

Original: (Makhzani & Frey, 2014; Ahmad & Scheinkman, 2019; van den Oord et al., 2017; Fedus et al., 2021).
Revised: (Makhzani & Frey, 2014; Ahmad & Scheinkman, 2019; van den Oord et al., 2017).

Original: We instead extend the elegant solution of Gozel & Gerstner (2021) by leveraging the inhibitory interneuron we have already introduced.
Revised: We instead extend the elegant solution of Gozel & Gerstner (2021) by leveraging the inhibitory interneuron we have already introduced.

Original: After neurogenesis, when neuronal dendrites are randomly initialized, neurons are excited by the inhibitory GABA neurotransmitter (see Appendix B.2).
Revised: After neurogenesis, when neuronal dendrites are randomly initialized, neurons are excited by the inhibitory GABA neurotransmitter (see App. B.2).

Original: This means that the same inhibitory interneuron used to enforce competition via Top-K inhibition at first creates cooperation by exciting every neuron, allowing them all to learn.
Revised: This means that the same inhibitory interneuron used to enforce competition via Top-K inhibition at first creates cooperation by exciting every neuron, allowing them all to learn.

Original: As a result, every neuron first converges onto the data manifold at which point it is inhibited by GABA and Top-K competition begins, forcing each neuron to specialize and tile the data manifold.
Revised: As a result, every neuron first converges onto the data manifold at which point it is inhibited by GABA and Top-K competition begins, forcing each neuron to specialize and tile the data manifold.

Original: We present the full GABA switch implementation in Appendix B.3 that is the most biologically plausible.
Revised: We present the full GABA switch implementation in App. B.3 that is the most biologically plausible.

Original: However, using the positive weight constraint that makes all activations positive, we found a simpler approximation that gives the same performance.
Revised: However, using the positive weight constraint that makes all activations positive, we found a simpler approximation that gives the same performance.

Original: This is by linearly annealing our k value from the maximum number of neurons in the layer down to our desired k .
Revised: This is by linearly annealing our k value from the maximum number of neurons in the layer down to our desired k .

Original: A similar approach without the biological justification is used by Makhzani & Frey (2014) but they zero out all values that are not the k most active and keep the remaining k untouched.
Revised: A similar approach without the biological justification is used by Makhzani & Frey (2014) but they zero out all values that are not the k most active and keep the remaining k untouched.

Original: Motivated by our inhibitory interneuron, we instead subtract the k +1 -th activation from all neurons that remain on.
Revised: Motivated by our inhibitory interneuron, we instead subtract the k + 1 -th activation from all neurons that remain on.

Original: This change leads to a significant boost in continual learning (see Table 2 and Appendix B.3).
Revised: This change leads to a significant boost in continual learning (see Table 2 and App. B.3).

Original: Formally, letting the neuron activations pre inhibition be a :=
Revised: Formally, letting the neuron activations pre inhibition be a :=

Original: X Ta ξ and a ∗ post inhibition:
Revised: X Ta ξ and a ∗ post inhibition:

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + is the ReLU operation, ⌊·⌋ is the floor operator to ensure k t is an integer, and descendingsort( · ) sorts the neuron activations in descending order to find the k + 1 -th largest activation.
Revised: where [ · ] + is the ReLU operation, ⌊·⌋ is the floor operator to ensure k t is an integer, and descendingsort( · ) sorts the neuron activations in descending order to find the k + 1 -th largest activation.

Original: E t is an integer representing the current training epoch and subscript t denotes that k t and E t change over time.
Revised: E t is an integer representing the current training epoch and subscript t denotes that k t and E t change over time.

Original: Hyperparameter s sets the number of epochs for k to go from its starting value k max , to its target value k target .
Revised: Hyperparameter s sets the number of epochs for k to go from its starting value k max , to its target value k target .

Original: When k t = k target our approximated GABA switch is fully inhibitory for every neuron.
Revised: When k t = k target our approximated GABA switch is fully inhibitory for every neuron.

Original: An algorithm box in Appendix A.1 summarizes how each SDMLP component functions.
Revised: An algorithm box in App. A.1 summarizes how each SDMLP component functions.

Original: The Stale Momentum Problem - We discovered that even using the GABA switch, some optimizers continue killing off a large fraction of neurons, to the detriment of continual learning.
Revised: The Stale Momentum Problem - We discovered that even using the GABA switch, some optimizers continue killing off a large fraction of neurons, to the detriment of continual learning.

Original: Investigating why, we have coined the “stale momentum” problem where optimizers that utilize some form of momentum (especially Adam and RMSProp) fail to compute an accurate moving average of previous gradients in the sparse activation setting (Kingma & Ba, 2015; Geoffrey Hinton, 2012).
Revised: Investigating why, we have coined the “stale momentum” problem where optimizers that utilize some form of momentum (especially Adam and RMSProp) fail to compute an accurate moving average of previous gradients in the sparse activation setting (Kingma & Ba, 2015; Geoffrey Hinton, 2012).

Original: Not only will these momentum optimizers update inactive neurons not in the Top-K, but also explode gradient magnitudes when neurons become activated after a period of quiescence.
Revised: Not only will these momentum optimizers update inactive neurons not in the Top-K, but also explode gradient magnitudes when neurons become activated after a period of quiescence.

Original: We explain and investigate stale momenta further in Appendix D and use SGD without momentum as our solution.
Revised: We explain and investigate stale momenta further in App. D and use SGD without momentum as our solution.

Original: Additional Modifications - There are five smaller discrepancies (expanded upon in Appendix A.5) between SDM and MLPs: (i) using rate codes to avoid non differentiable binary activations; (ii) using L 2 normalization of inputs and weights as an approximation to contrast encoding and heterosynapticity (Sterling & Laughlin, 2015; Rumelhart & Zipser, 1985; Tononi & Cirelli, 2014); (iii) removing bias terms which assume a tonic baseline firing rate not present in cerebellar granule cells (Powell et al., 2015; Giovannucci et al., 2017); (iv) using only positive (excitatory) weights that respect Dale’s Principle (Dale, 1935); (v) using backpropagation as a more efficient (although possibly non-biological (Lillicrap et al., 2020)) implementation of Hebbian learning rules (Krotov & Hopfield, 2019; Sengupta et al., 2018; Gozel & Gerstner, 2021; Millidge et al., 2020).
Revised: Additional Modifications - There are five smaller discrepancies (expanded upon in App. A.5) between SDM and MLPs: (i) using rate codes to avoid non differentiable binary activations; (ii) using L 2 normalization of inputs and weights as an approximation to contrast encoding and heterosynapticity (Sterling & Laughlin, 2015; Rumelhart & Zipser, 1985; Tononi & Cirelli, 2014); (iii) removing bias terms which assume a tonic baseline firing rate not present in cerebellar granule cells (Powell et al., 2015; Giovannucci et al., 2017); (iv) using only positive (excitatory) weights that respect Dale’s Principle (Dale, 1935); (v) using backpropagation as a more efficient (although possibly nonbiological (Lillicrap et al., 2020)) implementation of Hebbian learning rules (Krotov & Hopfield, 2019; Sengupta et al., 2018; Gozel & Gerstner, 2021).

Original:  SDM A VOIDS C ATASTROPHIC F ORGETTING
Revised:  SDM A VOIDS C ATASTROPHIC F ORGETTING

Original: Here we show that the proposed SDMLP architecture results in strong and organic continual learning.
Revised: Here we show that the proposed SDMLP architecture results in strong and organic continual learning.

Original: In modifying only the model architecture, our approach is highly compatible with other continual learning strategies such as regularization of important weights (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017) and memory replay (Zhang et al., 2021; Hsu et al., 2018), both of which the brain is likely to use in some fashion.
Revised: In modifying only the model architecture, our approach is highly compatible with other contin- ual learning strategies such as regularization of important weights (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017) and memory replay (Zhang et al., 2021; Hsu et al., 2018), both of which the brain is likely to use in some fashion.

Original: We show that combinations of SDMLP with weight regularization are positive sum and don’t explore memory replay.
Revised: We show that combinations of SDMLP with weight regularization are positive sum and don’t explore memory replay.

Original: Experimental Setup - Trying to make the continual learning setting as realistic as possible, we use Split CIFAR10 in the class incremental setting with pretraining on ImageNet (Russakovsky et al., 2015).
Revised: Experimental Setup - Trying to make the continual learning setting as realistic as possible, we use Split CIFAR10 in the class incremental setting with pretraining on ImageNet (Russakovsky et al., 2015).

Original: This splits CIFAR10 into disjoint subsets that each contain two of the classes.
Revised: This splits CIFAR10 into disjoint subsets that each contain two of the classes.

Original: For example the first data split contains classes 5 and 2 the second split contains classes 7 and 9, etc.
Revised: For example the first data split contains classes 5 and 2 the second split contains classes 7 and 9, etc.

Original: CIFAR is more complex than MNIST and captures real-world statistical properties of images.
Revised: CIFAR is more complex than MNIST and captures real-world statistical properties of images.

Original: The class incremental setting is more difficult than incremental task learning because predictions are made for every CIFAR class instead of just between the two classes in the current task (Hsu et al., 2018; Farquhar & Gal, 2018).
Revised: The class incremental setting is more difficult than incremental task learning because predictions are made for every CIFAR class instead of just between the two classes in the current task (Hsu et al., 2018; Farquhar & Gal, 2018).

Original: Pretraining on ImageNet enables learning general image statistics and is when the GABA switch happens, allowing neurons to specialize and spread across the data manifold.
Revised: Pretraining on ImageNet enables learning general image statistics and is when the GABA switch happens, allowing neurons to specialize and spread across the data manifold.

Original: In the main text, we present results where our ImageNet32 and CIFAR datasets have been compressed into 256 dimensional latent embeddings taken from the last layer of a frozen ConvMixer that was pre-trained on ImageNet32 (step #1 of our training regime in Fig. 2)
Revised: In the main text, we present results where our ImageNet32 and CIFAR datasets have been compressed into 256 dimensional latent embeddings taken from the last layer of a frozen ConvMixer that was pre-trained on ImageNet32 (step #1 of our training regime in Fig. 2)

Original: (Trockman & Kolter, 2022; Russakovsky et al., 2015).
Revised: (Trockman & Kolter, 2022; Russakovsky et al., 2015).

Original: Method Neurons k Val. Acc.
Revised: [Table]

Original: SDMLP 10K0.
Revised: SDMLP+MAS 10K

Original: FlyModel 10K 0.

Original: MAS 10K NA 0. SI 10K NA 0.
Revised: [Table]

Original: ReLU 10K NA 0. EWC 10K
Revised: SDMLP+EWC 10K

Original: NA 0.

Original: SDMLP+MAS 10K 0.

Original: SDMLP+EWC 10K 0.

Original: Oracle 10K NA 0. the full CIFAR10 dataset. All results are the average of 5 random task splits.
Revised: [Table]

Original: Using the image embeddings, we pre-train our models on ImageNet (step #2 of Fig. 2).
Revised: Using the image embeddings, we pre-train our models on ImageNet (step #2 of Fig. 2).

Original: We then reset X v that learns class labels and switch to continual learning on Split CIFAR10, training every model for 2,000 epochs on each of the five data splits (step #3 of Fig. 2).
Revised: We then reset X v that learns class labels and switch to continual learning on Split CIFAR10, training every model for 2,000 epochs on each of the five data splits (step #3 of Fig. 2).

Original: We use such a large number of epochs to encourage forgetting of previous tasks.
Revised: We use such a large number of epochs to encourage forgetting of previous tasks.

Original: We test model performance on both the current data split and every data split seen previously to assess forgetting.
Revised: We test model performance on both the current data split and every data split seen previously to assess forgetting.

Original: The CIFAR dataset is split into disjoint sets using five different random seeds to ensure our results are independent of both data split ordering and the classes each split contains.
Revised: The CIFAR dataset is split into disjoint sets using five different random seeds to ensure our results are independent of both data split ordering and the classes each split contains.

Original: We use preprocessed image embeddings because single hidden layer MLPs struggle to learn directly from image pixels.
Revised: We use preprocessed image embeddings because single hidden layer MLPs struggle to learn directly from image pixels.

Original: This preprocessing is also representative of the visual processing stream that compresses images used by deeper brain regions (Pisano et al., 2020;
Revised: This preprocessing is also representative of the visual processing stream that compresses images used by deeper brain regions (Pisano et al., 2020;

Original: Li et al., 2020; Yamins et al., 2014).
Revised: Li et al., 2020; Yamins et al., 2014).

Original: However, we consider two ablationsto our training regime that shift the validation accuracies but not the rank ordering of the models or conclusions drawn: (i) Removing the ConvMixer embeddings and instead training directly on image pixels (removing step #1 of Fig.
Revised: However, we consider two ablations that shift the validation accuracies but not the rank ordering of the models or conclusions drawn: (i) Removing the ConvMixer embeddings and instead training directly on image pixels (removing step #1 of Fig.

Original: 2, Appendix
Revised: 2, App.

Original: E.1); (ii) Testing continual learning without any ImageNet32 pre-training (removing step #2 of Fig.
Revised: E.1) ; (ii) Testing continual learning without any ImageNet32 pre-training (removing step #2 of Fig.

Original: 2, Appendix E.2).
Revised: 2, App. E.2).

Original: We also run experiments for CIFAR100 (Appendix F.2), MNIST (Appendix F.3) and FashionMNIST (Appendix F.4).
Revised: We also run experiments for CIFAR100 (App. F.2), MNIST (App. F.3) and FashionMNIST (App. F.4).

Original: Model Parameters and Baselines - All models are MLPs with 1,000 neurons in a single hidden layer unless otherwise indicated.
Revised: Model Parameters and Baselines - All models are MLPs with 1,000 neurons in a single hidden layer unless otherwise indicated.

Original: When using the Top-K activation function, we set k target = 10 and also present k target = 1 .
Revised: When using the Top-K activation function, we set k target = 10 and also present k target = 1 .

Original: We tested additional k values and suggest how to choose the best k target value in Appendix C.2.
Revised: We tested additional k values and suggest how to choose the best k target value in App. C.2.

Original: Because the k values considered are highly sparse – saving on FLOPs and memory consumption – we also evaluate the 10,000 neuron setting which improves SDM and the FlyModel continual learning abilities in particular.
Revised: Because the k values considered are highly sparse – saving on FLOPs and memory consumption – we also evaluate the 10,000 neuron setting which improves SDM and the FlyModel continual learning abilities in particular.

Original: The simplest baselines we implement are the ReLU activation function, L2 regularization, and
Revised: The simplest baselines we implement are the ReLU activation function, L2 regularization, and

Original: Dropout, that are all equally organic in not using any task information (Goodfellow et al., 2014).
Revised: Dropout, that are all equally organic in not using any task information (Goodfellow et al., 2014).

Original: We also compare to the popular regularization based approaches Elastic Weight Consolidation (EWC),
Revised: We also compare to the popular regularization based approaches Elastic Weight Consolidation (EWC),

Original: Memory Aware Synapses (MAS), and Synaptic Intelligence (SI) (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017; Smith et al., 2022).
Revised: Memory Aware Synapses (MAS), and Synaptic Intelligence (SI) (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017; Smith et al., 2022).

Original: These methods infer model weights that are important for each task and penalize updating them by using a regularization term in the loss.
Revised: These methods infer model weights that are important for each task and penalize updating them by using a regularization term in the loss.

Original: To track each parameter, this requires at least doubling memory consumption plus giving task boundaries and requiring a coefficient for the loss term.
Revised: To track each parameter, this requires at least doubling memory consumption plus giving task boundaries and requiring a coefficient for the loss term.

Original: In the class incremental learning setting, it has been shown that these regularization methods catastrophically forget (Hsu et al., 2018; Gurbuz & Dovrolis, 2022).
Revised: In the class incremental learning setting, it has been shown that these regularization methods catastrophically forget (Hsu et al., 2018; Gurbuz & Dovrolis, 2022).

Original: However, we found these results to be inaccurate and get better performance through careful hyperparameter tuning.
Revised: However, we found these results to be inaccurate and get better performance through careful hyperparameter tuning.

Original: This included adding a β coefficient to the softmax outputs of EWC and SI that alleviates the problem of vanishing gradients (see Appendix G.1).We are unaware of this simple modification being used in prior literature but use it to make our baselines more challenging.
Revised: This included adding a β coefficient to the softmax outputs of EWC and SI that alleviates the problem of vanishing gradients (see App. G.1).We are unaware of this simple modification being used in prior literature but use it to make our baselines more challenging.

Original: We also compare to related work that emphasizes biological plausibility and sparsity.
Revised: We also compare to related work that emphasizes biological plausibility and sparsity.

Original: The FlyModel performs very well and also requires no task information
Revised: The FlyModel performs very well and also requires no task information

Original: (Shen et al., 2021).
Revised: (Shen et al., 2021).

Original: Specific training parameters used for this model can be found in Appendix G.2.
Revised: Specific training parameters used for this model can be found in App. G.2.

Original: Additionally, we test Active Dendrites (Iyer et al., 2022) and NISPA (Neuro-Inspired Stability-Plasticity Adaptation) (Gurbuz & Dovrolis, 2022).
Revised: Additionally, we test Active Dendrites (Iyer et al., 2022) and NISPA (Neuro-Inspired Stability-Plasticity Adaptation) (Gurbuz & Dovrolis, 2022).

Original: However, these models both use the easier task incremental setting and failed to generalize well to the class incremental setting, especially Active Dendrites, on even the simplest Split MNIST dataset (Appendix F.3).
Revised: However, these models both use the easier task incremental setting and failed to generalize well to the class incremental setting, especially Active Dendrites, on even the simplest Split MNIST dataset (App. F.3).

Original: The failure for task incremental methods to apply in a class incremental setting has been documented before (Farquhar & Gal, 2018).
Revised: The failure for task incremental methods to apply in a class incremental setting has been documented before (Farquhar & Gal, 2018).

Original: These algorithms are also not organic, needing task labels and have significantly more complex, less biologically plausible implementations.
Revised: These algorithms are also not organic, needing task labels and have significantly more complex, less biologically plausible implementations.

Original: We considered a number of additional baselines but found that all were lacking existing results for the class incremental setting and often lacked publicly available codebases.
Revised: We considered a number of additional baselines but found that all were lacking existing results for the class incremental setting and often lacked publicly available codebases.

Original: All of our code and training parameters can be found in our publicly available codebase: https://github.com/anon8371/AnonPaper1 .

Original: We validation we easier.

Original: 1 visualizes how SDMLP gradually forgets each task compared to catastrophic forgetting of the ReLU baseline.

Original: We use the average of 5 random seeds and error bars show standard error of the mean but the variance is small making it hard to see

Original: CIFAR10 Results - We present the best continual learning results of each method in Fig.
Revised: CIFAR10 Results - We present the best continual learning results of each method in Fig.

Original: 3 and its corresponding Table 1.
Revised: 3 and its corresponding Table 1.

Original: SDMLP organically outperforms all regularization baselines, where its best validation accuracy is 71% compared to 67% for MAS.
Revised: SDMLP organically outperforms all regularization baselines, where its best validation accuracy is 71% compared to 67% for MAS.

Original: In the 1K neuron setting (shown in Tableof Appendix F.1) SDMLP with k = 1 ties with the FlyModel at 70%.
Revised: In the 1K neuron setting (shown in Table 5 of App. F.1) SDMLP with k = 1 ties with the FlyModel at 70%.

Original: In the 10K neuron setting, the FlyModel does much better achieving 82% versus 71% for SDMLP.
Revised: In the 10K neuron setting, the FlyModel does much better achieving 82% versus 71% for SDMLP.

Original: However, the combination of SDMLP and EWC leads to the strongest performance of 86%, barely forgetting any information compared to an oracle that gets 93% when trained on all of CIFAR10 simultaneously.
Revised: However, the combination of SDMLP and EWC leads to the strongest performance of 86%, barely forgetting any information compared to an oracle that gets 93% when trained on all of CIFAR10 simultaneously.

Original: SDMLP and the regularization based methods are complimentary and operate at different levels of abstraction.
Revised: SDMLP and the regularization based methods are complimentary and operate at different levels of abstraction.

Original: SDMLP learns what subset of neurons are important and the regularization method learns what subset of weights are important.
Revised: SDMLP learns what subset of neurons are important and the regularization method learns what subset of weights are important.

Original: Because the FlyModel combines fixed weights with
Revised: Because the FlyModel combines fixed weights with

Original: Top-K, it can be viewed as a more rigid form of the SDMLP and regularization method combined.
Revised: Top-K, it can be viewed as a more rigid form of the SDMLP and regularization method combined.

Original: The FlyModel also benefits from sparse weights making neurons more orthogonal to respond uniquely to different tasks.
Revised: The FlyModel also benefits from sparse weights making neurons more orthogonal to respond uniquely to different tasks.

Original: We attempted to integrate this weight sparsity into our SDMLP so that our neurons could learn the data manifold but be more constrained to a data subset, hypothetically producing less cross-talk and forgetting between tasks.
Revised: We attempted to integrate this weight sparsity into our SDMLP so that our neurons could learn the data manifold but be more constrained to a data subset, hypothetically producing less cross-talk and forgetting between tasks.

Original: Interestingly, initial attempts to prune weights either randomly or just the smallest from the ImageNet pretrained models resulted in catastrophic forgetting.
Revised: Interestingly, initial attempts to prune weights either randomly or just the smallest from the ImageNet pretrained models resulted in catastrophic forgetting.

Original: We leave more in depth explorations of weight sparsity to future work.
Revised: We leave more in depth explorations of weight sparsity to future work.

Original: Ablations - We ablate components of our SDMLP to determine which contribute to continual learning in Table 2.
Revised: Ablations - We ablate components of our SDMLP to determine which contribute to continual learning in Table 2.

Original: Deeper analysis of why each ablation fails is provided in Appendix Hbut we highlight the two most important here: First, the Top-K activation function prevents more than k neurons from updating for any given input, restricting the number of neurons that can forget information.
Revised: Deeper analysis of why each ablation fails is provided in App. H we highlight the two most important here: First, the Top-K activation function prevents more than k neurons from updating for any given input, restricting the number of neurons that can forget information.

Original: A small subset of the neurons will naturally specialize to the new data classes and continue updating towards it, protecting the remainder of the network from being overwritten.
Revised: A small subset of the neurons will naturally specialize to the new data classes and continue updating towards it, protecting the remainder of the network from being overwritten.

Original: This theory is supported by results in Appendix C.2 where the smaller k is, the less catastrophic forgetting occurs.
Revised: This theory is supported by results in App. C.2 where the smaller k is, the less catastrophic forgetting occurs.

Original: We provide further support for the importance of k by showing the number of neurons activated when learning each task in Appendix H.
Revised: We provide further support for the importance of k by showing the number of neurons activated when learning each task in App. H.

Original: Second, the L 2 normalization constraint and absence of a hidden layer bias term are crucial to keeping all neurons on the L 2 hypersphere, ensuring they all participate democratically.
Revised: Second, the L 2 normalization constraint and absence of a hidden layer bias term are crucial to keeping all neurons on the L 2 hypersphere, ensuring they all participate democratically.

Original: Without both these constraints, but still using the Top-K activation function, we found that a small subset of the neurons become “greedy” in having a larger weight norm or bias term and always out-compete the other neurons.
Revised: Without both these constraints, but still using the Top-K activation function, we found that a small subset of the neurons become “greedy” in having a larger weight norm or bias term and always out-compete the other neurons.

Original: This results in a small number of neurons updating for every new task and catastrophically forgetting the previous one as shown in the Table 2 ablations.
Revised: This results in a small number of neurons updating for every new task and catastrophically forgetting the previous one as shown in the Table 2 ablations.

Original: Evidence of manifold tiling is shown in Fig.
Revised: Evidence of manifold tiling is shown in Fig.

Original: 6 with a UMAP (McInnes & Healy, 2018) plot fit on the SDM weights (orange) that were pre-trained on ImageNet32 embeddings.
Revised: 6 with a UMAP (McInnes & Healy, 2018) plot fit on the SDM weights (orange) that were pre-trained on ImageNet32 embeddings.

Original: We project the embedded CIFAR training data (blue) to show that pre-training has given SDM subnetworks that cover the manifold of general image statistics contained in ImageNet32.
Revised: We project the embedded CIFAR training data (blue) to show that pre-training has given SDM subnetworks that cover the manifold of general image statistics contained in ImageNet32.

Original: This manifold tiling is in sharp contrast to the other methods shown in Appendix H and enables SDM to avoid catastrophic forgetting.
Revised: This manifold tiling is in sharp contrast to the other methods shown in App. H and enables SDM to avoid catastrophic forgetting.

Original: As further evidence of manifold tiling, we show in the same appendix that SDM weights are often highly interpretable when trained directly on CIFAR10 images.
Revised: As further evidence of manifold tiling, we show in the same appendix that SDM weights are often highly interpretable when trained directly on CIFAR10 images.

Original: As an additional ablation to our SDMLP defined by Eq.
Revised: As an additional ablation to our SDMLP defined by Eq.

Original: 2, we find that introducing a bias term in the output layer breaks continual learning.
Revised: 2, we find that introducing a bias term in the output layer breaks continual learning.

Original: This is because the model assigns large bias values to the classes within the current task over any classes in previous tasks, giving the appearance of catastrophic forgetting.
Revised: This is because the model assigns large bias values to the classes within the current task over any classes in previous tasks, giving the appearance of catastrophic forgetting.

Original: Interestingly, this effect only applies to SDM; modifying all of our other baselines by removing the output layer bias term fails to affect their continual learning performance.
Revised: Interestingly, this effect only applies to SDM; modifying all of our other baselines by removing the output layer bias term fails to affect their continual learning performance.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Setting out to implement SDM as an MLP, we introduce a number of modifications resulting in a model capable of continual learning.
Revised: Setting out to implement SDM as an MLP, we introduce a number of modifications resulting in a model capable of continual learning.

Original: These modifications are biologically inspired, incorporating components of cerebellar neurobiology that are collectively necessary for strong continual learning.
Revised: These modifications are biologically inspired, incorporating components of cerebellar neurobiology that are collectively necessary for strong continual learning.

Original:  Our new solution to the dead neuron problem and identification of the stale momentum problem are key to our results and may further generalize to other sparse neural network architectures.[Table]
Revised: [Table] Our new solution to the dead neuron problem and identification of the stale momentum problem are key to our results and may further generalize to other sparse neural network architectures.

Original: Being able to write SDM as an MLP with minor modifications is interesting in light of SDM’s connection to Transformer Attention (Bricken & Pehlevan, 2021).
Revised: Being able to write SDM as an MLP with minor modifications is interesting in light of SDM’s connection to Transformer Attention (Bricken & Pehlevan, 2021).

Original: This link converges with work showing that Transformer MLP layers perform associative memory-like operations that approximate
Revised: This link converges with work showing that Transformer MLP layers perform associative memory-like operations that approximate

Original: Top-K by showing up to 90% activation sparsity in later layers (Geva et al., 2020; Sukhbaatar et al., 2019; Jaszczur et al., 2021; Nelson et al., 2022; Meng et al., 2022).
Revised: Top-K by showing up to 90% activation sparsity in later layers (Geva et al., 2020; Sukhbaatar et al., 2019; Jaszczur et al., 2021; Nelson et al., 2022; Meng et al., 2022).

Original: Viewing both Attention and
Revised: Viewing both Attention and

Original: MLPs through the lens of SDM presents their tradeoffs: Attention operates on patterns in the model’s current receptive field.
Revised: MLPs through the lens of SDM presents their tradeoffs: Attention operates on patterns in the model’s current receptive field.

Original: This increases the fidelity of both the write and read operations because patterns do not need to rely on neurons that poorly approximate their original address location and store values in a noisy superposition.
Revised: This increases the fidelity of both the write and read operations because patterns do not need to rely on neurons that poorly approximate their original address location and store values in a noisy superposition.

Original: However, in the MLP setting, where patterns are stored in and read from neurons, an increase in noise is traded for being able to store patterns beyond the current receptive field.
Revised: However, in the MLP setting, where patterns are stored in and read from neurons, an increase in noise is traded for being able to store patterns beyond the current receptive field.

Original: This can allow for learning statistical regularities in the patterns’ superpositions to represent useful prototypes, likely benefiting generalization (Irie et al., 2022).
Revised: This can allow for learning statistical regularities in the patterns’ superpositions to represent useful prototypes, likely benefiting generalization (Irie et al., 2022).

Original: Limitations - Our biggest limitation remains ensuring that SDM successfully avoids the dead neuron problem and tiles the data manifold in order to continually learn.
Revised: Limitations - Our biggest limitation remains ensuring that SDM successfully avoids the dead neuron problem and tiles the data manifold in order to continually learn.

Original: Settingthe k targetvalue and s decay rate remain difficult and data manifold dependent (Appendix C.2).
Revised: Setting k target and s decay rate remain difficult and data manifold dependent (App. C.2).

Original: Avoiding dead neurons currently requires training SDM on a static data manifold, whether it is image pixels or a fixed embedding.
Revised: Avoiding dead neurons currently requires training SDM on a static data manifold, whether it is image pixels or a fixed embedding.

Original: Initial experiments jointly training SDM modules either interleaved throughout a ConvMixer or placed at the end resulted in many dead neurons and failure to continually learn.
Revised: Initial experiments jointly training SDM modules either interleaved throughout a ConvMixer or placed at the end resulted in many dead neurons and failure to continually learn.

Original: We believe this is in large part due to the manifold continuing to change over time (Appendix C.2).
Revised: We believe this is in large part due to the manifold continuing to change over time.

Original: Constraining neurons to exist on the data manifold and only allowing a highly sparse subset to fire is also suboptimal for maximizing classification accuracy in non continual learning settings.
Revised: Constraining neurons to exist on the data manifold and only allowing a highly sparse subset to fire is also suboptimal for maximizing classification accuracy in non continual learning settings.

Original: Making k sufficiently large can alleviate this problem but at the cost of continual learning abilities.
Revised: Making k sufficiently large can alleviate this problem but at the cost of continual learning abilities.

Original: However, the ability for hardware accelerators to leverage sparsity should allow for networks to become wider (increasing r ), rather than decreasing k , and help alleviate this issue (Wang, 2020; Gale et al., 2020).
Revised: However, the ability for hardware accelerators to leverage sparsity should allow for networks to become wider (increasing r ), rather than decreasing k , and help alleviate this issue (Wang, 2020; Gale et al., 2020).

Original: Conclusion - We have shown that SDM, when given the ability to model correlated data manifolds in ways that respect cerebellar neurobiology, is a strong continual learner.
Revised: Conclusion - We have shown that SDM, when given the ability to model correlated data manifolds in ways that respect cerebellar neurobiology, is a strong continual learner.

Original: This result is striking because every component of the SDMLP, including L 2 normalization, the GABA switch, and momentum free optimization are necessary for these continual learning results.
Revised: This result is striking because every component of the SDMLP, including L 2 normalization, the GABA switch, and momentum free optimization are necessary for these continual learning results.

Original: Beyond setting a new “organic” baseline for continual learning, we help establish how to train sparse models in the deep learning setting through solutions to the dead neuron and stale momentum problems.
Revised: Beyond setting a new “organic” baseline for continual learning, we help establish how to train sparse models in the deep learning setting through solutions to the dead neuron and stale momentum problems.

Original: More broadly, this work expands the neurobiological mapping of SDM to cerebellar circuitry and relates it to
Revised: More broadly, this work expands the neurobiological mapping of SDM to cerebellar circuitry and relates it to

Original: MLPs, deepeningits links toboth the brain and deep learning. 
Revised: MLPs, deepening links to the brain and deep learning. A CKNOWLEDGEMENTS

Original: Appendix Table of Contents
Revised: Thanks to Dr. Beren Millidge, Joe Choo-Choy, Miles Turpin, Blake Bordelon, Stephen Casper, Dr. Mengmi Zhang, Dr. Tomaso Poggio and Dr. Pentti Kanerva for providing invaluable inspiration, discussions and feedback. We would also like to thank the open source software contributors that helped make this research possible, including but not limited to: Numpy, Pandas, Scipy, Matplotlib, PyTorch, and Anaconda.

Original: [Figure] SDM A.1 SDM Training Algorithm
Revised: • Trenton Bricken conceived of the project and theory, conducted all experiments, and wrote the paper with suggestions from co-authors.

Original: A.2 SDM Biological Plausibility A.3 Learning SDM Neuron Addresses with the Top-K Activation Function A.4 Why SDM Originally Required Fixed Neuron Addresses A.5 Additional Modifications to SDM
Revised: • Xander Davies participated in discussions, contributed to early investigations of Dead Neurons and Stale Momentum, and reviewed related work.

Original: A.6 SDM Write Operation Relation to MLP Backpropagation A.7 Rate Activations Maintain SDM’s Approximation to Transformer Attention
Revised: • Deepak Singh reviewed robustness and Top-K related work and participated in discussions. • Dmitry Krotov advised on the continual learning experiments, Top-K learning theory, and relations to associative memory models.

Original: B.1 Biologically Implausible Solutions to the Dead Neuron Problem . B.2 GABA Switch Biology .
Revised: • Gabriel Kreiman supervised the project providing guidance throughout on theory and experiments.

Original: B.3 GABA Switch Practical Considerations -K
Revised: F Additional Datasets

Original: C.1 Inhibitory Interneurons Approximate Top-K [Table] [Table] [Table]
Revised: F.1 CIFAR10 Extra Fig. and Table . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table]
Revised: F.2 CIFAR100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.3 Split MNIST . .

Original: [Table] [Table]
Revised: F

Original: [Table]
Revised: [Table]

Original: Figure and Table . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 4 Split FashionMNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table] [Table] [Table]
Revised: G

Original: [Table]
Revised: Beta Consolidation Synaptic Intelligence . . .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H Investigating Differences in Continual Learning Abilities
Revised: H Investigating Differences in Continual Learning Abilities

Original: [Equation]
Revised: A SDM A. SDM T RAINING A LGORITHM

Original: Here we present in full the training loop used for the SDMLP.
Revised: Here we present in full the training loop used for the SDMLP.

Revised: Input: Model weights: ( X a , X v ) , Tasks: ( T 1 , . .

Revised: T n ) , Top-K params: k max , k target , Train steps: s

Revised: Output: Trained model weights: ( X a , X v )

Original: X v ← LayerInit ( X v ) //
Revised: X v ← LayerInit ( X v ) //

Original: Reset neuron output weights for i in (1,...,n) do
Revised: Reset neuron output weights for i in (1,...,n) do

Original: X , Y ← T i // Obtain data and class labels for current task for j in (1,...,s) do x , y ∼ X , Y //
Revised: X , Y ← T i // Obtain data and class labels for current task for j in (1,...,s) do x , y ∼ X , Y //

Original: Sample a data point from the task x ← x / || x
Revised: Sample a data point from the task x ← x / || x

Original: || //
Revised: || //

Original: L2 normalize the data a ← X Ta x // Get neuron activations a ← Top-K ( a ) //
Revised: L2 normalize the data a ← X Ta x // Get neuron activations a ← Top-K ( a ) //

Original: Apply Top-K Eq.
Revised: Apply Top-K Eq.

Original: ˆ y ← X Tv a // Get model predictions ∇ θ X a , ∇ θ X v ← ComputeGradients (ˆ y , y ) //
Revised: ˆ y ← X Tv a // Get model predictions ∇ θ X a , ∇ θ X v ← ComputeGradients (ˆ y , y ) //

Original: Compute Loss and Gradients
Revised: Compute Loss and Gradients

Original: X a , X v ← GradientStep ( ∇ θ X a , ∇ θ X v ) //
Revised: X a , X v ← GradientStep ( ∇ θ X a , ∇ θ X v ) //

Original: Update model weights
Revised: Update model weights

Original: X a , X v ← [ X a ] + , [ X v ] + // Clamp all weights to be positive
Revised: X a , X v ← [ X a ] + , [ X v ] + // Clamp all weights to be positive

Original: X a ← X a / || X
Revised: X a ← X a / || X

Original: a || //
Revised: a || //

Original: L2 normalize the neuron addresses end endreturn X a , X v
Revised: L2 normalize the neuron addresses end end return X a , X v

Original: A. SDM B IOLOGICAL P LAUSIBILITY
Revised: A. SDM B IOLOGICAL P LAUSIBILITY

Original: Here we summarize the biological foundations of SDM.[-
-]
Revised: Here we summarize the biological foundations of SDM.[-
-]

Original: The biological plausibility of our new modifications to SDM that allow it to learn the data manifold and be implemented as a deep learning model can be found in Appendix A.5.
Revised: The biological plausibility of our new modifications to SDM that allow it to learn the data manifold and be implemented as a deep learning model can be found in App. A.5.

Original: The biological plausibility of the GABA switch can be found in Appendix B.2. 
Revised: The biological plausibility of the GABA switch can be found in App. B.2. Fig.

Original: During SDM read operations, this Purkinje cell can thus sum over index i of all activated neuron storage vectors and use its firing rate to represent the value of y i . 
Revised: 5 overlays SDM notation and operations on the cerebellar circuitry. Mossy Fibers represent incoming queries ξ and pattern addresses p a through their firing activity. Each Granule cell represents an SDM neuron x , with its dendrites as the neuron address vector x a and post-synaptic connections with Purkinje cells as the storage vector x v . In branching perpendicularly, each Granule cells axon (called a Parallel Fiber), forms contacts with thousands of Purkinje cells with many granule cells (Hoxha et al., 2016).

Revised: The strength of each contact represents a counter recording the value of a particular element in the neuron storage vector ( x v ) i , where i indexes this vector element.

Revised: During SDM read operations, this Purkinje cell can thus sum over index i of all activated neuron storage vectors and use its firing rate to represent the value of y i .

Original: During SDM write operations, the Climbing Fibers are used to update the Purkinje cell counter of each active neuron with the pattern value at this index position ( p v ) i .
Revised: During SDM write operations, the Climbing Fibers are used to update the Purkinje cell counter of each active neuron with the pattern value at this index position ( p v ) i .

Original: The most demanding part of SDM’s implementation in a biological circuit that is strikingly satisfied by the cerebellum is the three way interface and specificity that exists between Purkinje cells, Climbing Fibers, and Parallel Fibers.
Revised: The most demanding part of SDM’s implementation in a biological circuit that is strikingly satisfied by the cerebellum is the three way interface and specificity that exists between Purkinje cells, Climbing Fibers, and Parallel Fibers.

Original: This allows for
Revised: This allows for

Original: Granule cells to have storage vectors that are precisely written to and read from.
Revised: Granule cells to have storage vectors that are precisely written to and read from.

Original: We also provide
Revised: We also provide

Original: Figure 6 to give an additional perspective on the biological mapping.
Revised: Fig. 6 to give an additional perspective on the biological mapping.

Original: While our improved version of SDM assigns functionality to five different cell types in the cerebellar circuit, there is more of the circuitry to be mapped such as the Basket cells and Deep Cerebellar
Revised: While our improved version of SDM assigns functionality to five different cell types in the cerebellar circuit, there is more of the circuitry to be mapped such as the Basket cells and Deep Cerebellar

Original: Nuclei (Sezener et al., 2021).
Revised: Nuclei (Sezener et al., 2021).

Original: There is also additional neuroscientific evidence required to confirm that the cerebellum operates as SDM predicts, for example that granule cells fire sparsely (Kanerva, 1988; Bricken & Pehlevan, 2021; Giovannucci et al., 2017).
Revised: There is also additional neuroscientific evidence required to confirm that the cerebellum operates as SDM predicts, for example that granule cells fire sparsely (Kanerva, 1988; Bricken & Pehlevan, 2021; Giovannucci et al., 2017).

Original: Within the scope of our model, the weakest biological plausibility is how well Top-K actually approximates the Golgi inhibitory interneuron (Appendix C.1) and how L 2 normalization is implemented (Appendix A.5). 
Revised: Within the scope of our model, the weakest biological plausibility is how well Top-K actually approximates the Golgi inhibitory interneuron (App. C.1) and how L 2 normalization is implemented (App. A.5). as Fig. of (

Original: [Figure]
Revised: [Figure]

Original: A. L EARNING SDM N EURON A DDRESSES WITH THE T OP -K A CTIVATION F UNCTION
Revised: A. L EARNING SDM N EURON A DDRESSES WITH THE T OP -K A CTIVATION F UNCTION

Original: SDM was built on the assumption that its neuron addresses, X a , are randomly initialized and fixed in place (Kanerva, 1988).
Revised: SDM was built on the assumption that its neuron addresses, X a , are randomly initialized and fixed in place (Kanerva, 1988).

Original: This was for reasons of both biological plausibility and analytical tractability when determining the maximum memory capacity and convergence dynamics of the model (see
Revised: This was for reasons of both biological plausibility and analytical tractability when determining the maximum memory capacity and convergence dynamics of the model (see

Original: Appendix A.4).
Revised: App. A.4).

Original: However, these random and fixed neuron addresses will often not be within cosine similarity c of the real-world patterns existing on some lower dimensional manifold in the vector space.
Revised: However, these random and fixed neuron addresses will often not be within cosine similarity c of the real-world patterns existing on some lower dimensional manifold in the vector space.

Original: As a result, these neurons will often never be written to or read from.
Revised: As a result, these neurons will often never be written to or read from.

Original: Keeler (1988) outlined how SDM could both retain its theoretical results and allow for neuron addresses to learn the data manifold.
Revised: Keeler (1988) outlined how SDM could both retain its theoretical results and allow for neuron addresses to learn the data manifold.

Original: This is by replacing the fixed cosine similarity activation threshold c that activates a variable number of neurons, with a variable threshold that activates a fixed k neurons.
Revised: This is by replacing the fixed cosine similarity activation threshold c that activates a variable number of neurons, with a variable threshold that activates a fixed k neurons.

Original: Intuitively, with random neuron and pattern addresses, a fixed c would result in a pattern or query activating some k neurons in expectation, keeping neuron utilization constant.
Revised: Intuitively, with random neuron and pattern addresses, a fixed c would result in a pattern or query activating some k neurons in expectation, keeping neuron utilization constant.

Original: However, using the same fixed c with non-random addresses would vary the number of neurons being activated and utilized.
Revised: However, using the same fixed c with non-random addresses would vary the number of neurons being activated and utilized.

Original: Over-utilized neurons would store too many pattern values in their storage vector superposition, harming pattern fidelity.
Revised: Over-utilized neurons would store too many pattern values in their storage vector superposition, harming pattern fidelity.

Original: Instead, we can achieve the same constant neuron utilization as in the random address setting by varying c such that only k neurons are always activated.
Revised: Instead, we can achieve the same constant neuron utilization as in the random address setting by varying c such that only k neurons are always activated.

Original: How can c be dynamically adjusted to keep a constant k neurons active in a biologically plausible fashion?
Revised: How can c be dynamically adjusted to keep a constant k neurons active in a biologically plausible fashion?

Original: Via an inhibitory interneuron that creates a negative feedback loop: the more neurons that are active, the more activated the interneuron becomes, and the more it inhibits, keeping only k active at convergence.
Revised: Via an inhibitory interneuron that creates a negative feedback loop: the more neurons that are active, the more activated the interneuron becomes, and the more it inhibits, keeping only k active at convergence.

Original: 9 Sparsity inducing interneurons are ubiquitious across layers of cortical processing in the brain, particularly relevant to SDM is the cerebellar Golgi interneuron (Marr, 1969; Keeler, 1988; Paiton et al., 2020).
Revised: 10 Sparsity inducing interneurons are ubiquitious across layers of cortical processing in the brain, particularly relevant to SDM is the cerebellar Golgi interneuron (Marr, 1969; Keeler, 1988; Paiton et al., 2020).

Original: A. W HY SDM O RIGINALLY R EQUIRED F IXED N EURON A DDRESSES
Revised: A. W HY SDM O RIGINALLY R EQUIRED F IXED N EURON A DDRESSES

Original: In attempting to be biologically plausible, SDM was designed to respect Dale’s Principle, whereby a synaptic connection can be either excitatory or inhibitory but not transition between them (Dale, 1935).
Revised: In attempting to be biologically plausible, SDM was designed to respect Dale’s Principle, whereby a synaptic connection can be either excitatory or inhibitory but not transition between them (Dale, 1935).

Original: Using the original binary vector formulation, neurons could compute if they were near enough to an incoming pattern or query to be activated (within some Hamming distance threshold) through the following steps:
Revised: Using the original binary vector formulation, neurons could compute if they were near enough to an incoming pattern or query to be activated (within some Hamming distance threshold) through the following steps:

Original: Converting the neuron’s binary address to bipolar weights { +1 , − 1 } corresponding to being excitatory or inhibitory, respectively.
Revised: Converting the neuron’s binary address to bipolar weights { +1 , − 1 } corresponding to being excitatory or inhibitory, respectively.

Original: Activating those weights where the incoming query has a 1 in its address and summing over them.
Revised: Activating those weights where the incoming query has a 1 in its address and summing over them.

Original: This corresponds to how many 1s in the query and neuron address vectors agree – if the addresses match then an excitatory weight is activated, giving a +1, if they disagree then a negative weight is activated, giving a -1.
Revised: This corresponds to how many 1s in the query and neuron address vectors agree – if the addresses match then an excitatory weight is activated, giving a +1, if they disagree then a negative weight is activated, giving a -1.

Original: Rescaling the binary Hamming distance threshold if the neuron is activated by the total number of 1s in the neuron address.
Revised: Rescaling the binary Hamming distance threshold if the neuron is activated by the total number of 1s in the neuron address.

Original: See (Kanerva, 1988; 1993) for details beyond this summary.
Revised: See (Kanerva, 1988; 1993) for details beyond this summary.

Original: Formally, the binary neuron address can be converted into weights w i = ± 1 with:
Revised: Formally, the binary neuron address can be converted into weights w i = ± 1 with:

Original: [Equation]
Revised: [Equation]

Original: Where τ is used to denote a specific neuron address, the query vector is ξ , and the function to determine if the neuron should be activated is a ( · , · ) that returns a binary action potential.
Revised: Where τ is used to denote a specific neuron address, the query vector is ξ , and the function to determine if the neuron should be activated is a ( · , · ) that returns a binary action potential.

Original: We can write:
Revised: We can write:

Original: [Equation]
Revised: [Equation]

Original: where there is the action potential threshold d for the neuron to fire and I [ · ] is the indicator function.
Revised: where there is the action potential threshold d for the neuron to fire and I [ · ] is the indicator function.

Original: The interval of values for address decoding is the number of 1s in the address, [ − (cid:80) ni
Revised: The interval of values for address decoding is the number of 1s in the address, [ − (cid:80) ni

Original: [ x τa ] i , (cid:80) ni [ x τa ] i ] .
Revised: [ x τa ] i , (cid:80) ni [ x τa ] i ] .

Original: By adjusting our d value to account for the possible number of matches, we can implement the Hamming distance threshold for each neuron.
Revised: By adjusting our d value to account for the possible number of matches, we can implement the Hamming distance threshold for each neuron.

Original: The potential for changing the magnitude of neuron weights was considered but a change of their sign was banned due to Dale’s Principle.
Revised: The potential for changing the magnitude of neuron weights was considered but a change of their sign was banned due to Dale’s Principle.

Original: It was acknowledged in (Kanerva, 1988) that changing the magnitudes of the neuron weights results in a weighted sum where the matching of some bits in the addresses matters more than others for calculating if the neuron is activated.
Revised: It was acknowledged in (Kanerva, 1988) that changing the magnitudes of the neuron weights results in a weighted sum where the matching of some bits in the addresses matters more than others for calculating if the neuron is activated.

Original: However, the crucial feature is the sign of the weights, determining if there is a match or mismatch.
Revised: However, the crucial feature is the sign of the weights, determining if there is a match or mismatch.

Original: This sign change is what was banned by Dale’s Principle, making the neuron addresses fixed both to their sign and to their specific value by the use of binary vectors that cannot represent nuances in weight magnitude (Kanerva, 1988; 1993).
Revised: This sign change is what was banned by Dale’s Principle, making the neuron addresses fixed both to their sign and to their specific value by the use of binary vectors that cannot represent nuances in weight magnitude (Kanerva, 1988; 1993).

Original: However, the original SDM approach still in fact violated Dale’s Principle by mapping the binary neuron address to bipolar because of what this means for the pre-synaptic input neurons.
Revised: However, the original SDM approach still in fact violated Dale’s Principle by mapping the binary neuron address to bipolar because of what this means for the pre-synaptic input neurons.

Original: Consider the 3rd element of the input vector (either a pattern or a query) being on (a 1 value).
Revised: Consider the 3rd element of the input vector (either a pattern or a query) being on (a 1 value).

Original: The neuron that is active and represents this value will not have a mixture of both excitatory and inhibitory efferent connections with the SDM neuron addresses.
Revised: The neuron that is active and represents this value will not have a mixture of both excitatory and inhibitory efferent connections with the SDM neuron addresses.

Original: In the cerebellar mapping, mossy fibers represent the pattern/query inputs and release glutamate.
Revised: In the cerebellar mapping, mossy fibers represent the pattern/query inputs and release glutamate.

Original: This means they are always excitatory to granule cells and the 0s in a neuron address should correspond to having no weight (no dendritic connection) rather than a negative weight (inhibitory connection).
Revised: This means they are always excitatory to granule cells and the 0s in a neuron address should correspond to having no weight (no dendritic connection) rather than a negative weight (inhibitory connection).

Original: An additional issue with the original SDM formulation is that the neuron addresses are dense vectors; this is not the case for the granule cells that they are mapped onto in the cerebellum where each has only ∼ 4 dendrites (Litwin-Kumar et al., 2017)
Revised: An additional issue with the original SDM formulation is that the neuron addresses are dense vectors; this is not the case for the granule cells that they are mapped onto in the cerebellum where each has only ∼ 4 dendrites (Litwin-Kumar et al., 2017)

Original: (see (Jaeckel, 1989a) for a solution that makes the weights of SDM sparse).
Revised: (see (Jaeckel, 1989a) for a solution that makes the weights of SDM sparse).

Original: Taken together, these issues can be resolved by having the binary neuron addresses be sparse and staying binary rather than becoming bipolar.
Revised: Taken together, these issues can be resolved by having the binary neuron addresses be sparse and staying binary rather than becoming bipolar.

Original: We can then allow our neuron addresses to use positive real values, simultaneously allowing for changes in weight strengths and respecting Dale’s Principle.
Revised: We can then allow our neuron addresses to use positive real values, simultaneously allowing for changes in weight strengths and respecting Dale’s Principle.

Original: In this work, we use positive real values but do allow for our weights to be dense, leaving high degrees of weight sparsity to future exploration (Jaeckel, 1989a;b).
Revised: In this work, we use positive real values but do allow for our weights to be dense, leaving high degrees of weight sparsity to future exploration (Jaeckel, 1989a;b).

Original: A. A DDITIONAL M ODIFICATIONS TO SDM
Revised: A. A DDITIONAL M ODIFICATIONS TO SDM

Original: The five modifications made to SDM for it to be implemented in a deep learning framework and that differentiate it from a vanilla MLP are explained in full here.
Revised: The five modifications made to SDM for it to be implemented in a deep learning framework and that differentiate it from a vanilla MLP are explained in full here.

Original: These modifications are:
Revised: These modifications are:

Original: Using continuous instead of binary neuron activations 2.
Revised: Using continuous instead of binary neuron activations 2.

Original: L 2 normalization of inputs and weights 3.
Revised: L 2 normalization of inputs and weights 3.

Original: No bias term 4.
Revised: No bias term 4.

Original: Only positive weights 5.
Revised: Only positive weights 5.

Original: Backpropagation
Revised: Backpropagation

Original: Continuous Activations - SDM originally modelled neurons as having binary action potentials by using a Heaviside step function.
Revised: Continuous Activations - SDM originally modelled neurons as having binary action potentials by using a Heaviside step function.

Original: However, this is non-differentiable and we want to use backpropagation for training our model.
Revised: However, this is non-differentiable and we want to use backpropagation for training our model.

Original: This can be resolved by viewing a neuron’s action potentials over a time interval, referred to in the neuroscience literature as a rate code (Dayan & Abbott, 2001).
Revised: This can be resolved by viewing a neuron’s action potentials over a time interval, referred to in the neuroscience literature as a rate code (Dayan & Abbott, 2001).

Original: We believe this is compatible with SDM, whereby neurons with addresses closer to an incoming pattern or query will receive stronger stimulation and fire more action potentials by having more dendritic connections stimulated by excitatory neurotransmitters.
Revised: We believe this is compatible with SDM, whereby neurons with addresses closer to an incoming pattern or query will receive stronger stimulation and fire more action potentials by having more dendritic connections stimulated by excitatory neurotransmitters.

Original: We can represent neuronal activation as an expected firing rate with a real positive number a ∈ [0 , t ] where t is some maximum firing rate.
Revised: We can represent neuronal activation as an expected firing rate with a real positive number a ∈ [0 , t ] where t is some maximum firing rate.

Original: This may look like a rectified tanh function where, due to refractory periods, there are diminishing returns to more stimulation (Glorot et al., 2011).
Revised: This may look like a rectified tanh function where, due to refractory periods, there are diminishing returns to more stimulation (Glorot et al., 2011).

Original: However, because our neurons are already constrained by L 2 normalization to be in the cosine similarity bounds of [ − 1 , 1] we simply use a ReLU activation function.
Revised: However, because our neurons are already constrained by L 2 normalization to be in the cosine similarity bounds of [ − 1 , 1] we simply use a ReLU activation function.

Original: Implementing weighted read and write operations in our original SDM Eqs.
Revised: Implementing weighted read and write operations in our original SDM Eqs.

Original: 1 and 2, we would replace our binarizing function b ( · ) with a weight coefficient proportional to the distance between the input and neuron addresses.
Revised: 1 and 2, we would replace our binarizing function b ( · ) with a weight coefficient proportional to the distance between the input and neuron addresses.

Original: We show in Appendix A.7 that this modification has minimal impact on how SDM weights different patterns.
Revised: We show in App. A.7 that this modification has minimal impact on how SDM weights different patterns.

Original: This means it should have minimal effect on memory capacity and is still approximately exponential, maintaining its relationship to Transformer Attention (Bricken & Pehlevan, 2021).
Revised: This means it should have minimal effect on memory capacity and is still approximately exponential, maintaining its relationship to Transformer Attention (Bricken & Pehlevan, 2021).

Original: L 2 Normalization - SDM requires a valid distance metric to compute if neurons and patterns/queries are sufficiently close to each other to produce an activation.
Revised: L 2 Normalization - SDM requires a valid distance metric to compute if neurons and patterns/queries are sufficiently close to each other to produce an activation.

Original: We make SDM continuous so that it is differentiable by following (Bricken & Pehlevan, 2021) and replacing its original Hamming distance with cosine similarity.
Revised: We make SDM continuous so that it is differentiable by following (Bricken & Pehlevan, 2021) and replacing its original Hamming distance with cosine similarity.

Original: In SDM’s mapping to the cerebellum (outlined in Appendix A.2), mossy fibers represent input patterns/queries and granule cell dendrites represent neuron addresses.
Revised: In SDM’s mapping to the cerebellum (outlined in App. A.2), mossy fibers represent input patterns/queries and granule cell dendrites represent neuron addresses.

Original: It remains to be experimentally established if and how these specific cell types enforce L 2 normalization.
Revised: It remains to be experimentally established if and how these specific cell types enforce L 2 normalization.

Original: However, contrast encoding and heterosynapticity are both ubiquitous mechanisms that can approximate L 2 normalization for the mossy fiber activations and granule dendrites, respectively (Sterling & Laughlin, 2015;
Revised: However, contrast encoding and heterosynapticity are both ubiquitous mechanisms that can approximate L 2 normalization for the mossy fiber activations and granule dendrites, respectively (Sterling & Laughlin, 2015;

Original: Rumelhart & Zipser, 1985; Tononi & Cirelli, 2014).
Revised: Rumelhart & Zipser, 1985; Tononi & Cirelli, 2014).

Original: From a deep learning perspective, LayerNorm and BatchNorm are both used ubiquitously and can also be viewed as approximations to L 2 normalization (Ioffe & Szegedy, 2015; Ba et al., 2016).
Revised: From a deep learning perspective, LayerNorm and BatchNorm are both used ubiquitously and can also be viewed as approximations to L 2 normalization (Ioffe & Szegedy, 2015; Ba et al., 2016).

Original: No Bias Term - In order to have our neuron activations represent cosine similarities, we must remove the bias term.
Revised: No Bias Term - In order to have our neuron activations represent cosine similarities, we must remove the bias term.

Original: We also remove the bias term from the output layer so that outputs represent only a summation of the neuron storage vectors that are activated.
Revised: We also remove the bias term from the output layer so that outputs represent only a summation of the neuron storage vectors that are activated.

Original: The absence of both of these bias terms is clear in the SDM Eq.
Revised: The absence of both of these bias terms is clear in the SDM Eq.

Original: 2 compared to the MLP Eq.
Revised: 2 compared to the MLP Eq.

Original: Bias terms can be viewed biologically as representing a neuron activation threshold (if negative) or a baseline tonic firing rate (if positive).
Revised: Bias terms can be viewed biologically as representing a neuron activation threshold (if negative) or a baseline tonic firing rate (if positive).

Original: In SDM’s cerebellar mapping, granule cells that represent the neurons do not have a tonic baseline firing rate meaning that positive bias terms should not be allowed (Powell et al., 2015; Giovannucci et al., 2017).
Revised: In SDM’s cerebellar mapping, granule cells that represent the neurons do not have a tonic baseline firing rate meaning that positive bias terms should not be allowed (Powell et al., 2015; Giovannucci et al., 2017).

Original: Purkinje cell firing that represents the output layer is much more complex such that keeping or removing the bias term is hard to justify biologically
Revised: Purkinje cell firing that represents the output layer is much more complex such that keeping or removing the bias term is hard to justify biologically

Original: (Zeeuw, 2020) but we follow the SDM equation in also removing it.
Revised: (Zeeuw, 2020) but we follow the SDM equation in also removing it.

Original: While removing positive bias terms from the granule cells fits neurobiology, removing negative bias terms corresponding to an activation threshold is less justified.
Revised: While removing positive bias terms from the granule cells fits neurobiology, removing negative bias terms corresponding to an activation threshold is less justified.

Original: However, as it relates to learning dynamics and the ability for Top-K to still be approximated, the ordinary differential equations of (Gozel & Gerstner, 2021) (summarized in C.1), still maintain approximately k neurons firing while using activation thresholds.
Revised: However, as it relates to learning dynamics and the ability for Top-K to still be approximated, the ordinary differential equations of (Gozel & Gerstner, 2021) (summarized in C.1), still maintain approximately k neurons firing while using activation thresholds.

Original: This is because the fewer neurons that are firing, the less the inhibitory interneuron is activated, keeping more neurons active.
Revised: This is because the fewer neurons that are firing, the less the inhibitory interneuron is activated, keeping more neurons active.

Original: As a result, and in line with SDM Eq.
Revised: As a result, and in line with SDM Eq.

Original: 2, we make the simplifying assumption of not allowing for negative bias terms either, thus removing bias terms entirely.
Revised: 2, we make the simplifying assumption of not allowing for negative bias terms either, thus removing bias terms entirely.

Original: An additional benefit of SDM having no bias term, in conjunction with positive weights, is that all neurons will have positive activations by default.
Revised: An additional benefit of SDM having no bias term, in conjunction with positive weights, is that all neurons will have positive activations by default.

Original: This allows for k annealing instead of the GABA switch that is otherwise needed to inject positive current into each neuron when learning the data manifold (also discussed in Appendix B.3).
Revised: This allows for k annealing instead of the GABA switch that is otherwise needed to inject positive current into each neuron when learning the data manifold (also discussed in App. B.3).

Original: Having all neurons active also guarantees that there will always be at least
Revised: Having all neurons active also guarantees that there will always be at least

Original: k active neurons to use in the Top-K.
Revised: k active neurons to use in the Top-K.

Original: It is noteworthy that the removal of the bias terms is seeing a resurgence in state of the art models such as the 540B parameter PaLM Transformer language model (Chowdhery et al., 2022), which noted that removal of bias terms resulted in more stable training.
Revised: It is noteworthy that the removal of the bias terms is seeing a resurgence in state of the art models such as the 540B parameter PaLM Transformer language model (Chowdhery et al., 2022), which noted that removal of bias terms resulted in more stable training.

Original: Positive Weights - The connection between mossy fibers and granule cells is excitatory so the weights should be positive.
Revised: Positive Weights - The connection between mossy fibers and granule cells is excitatory so the weights should be positive.

Original: Allowing for both positive and negative weights would violate Dale’s
Revised: Allowing for both positive and negative weights would violate Dale’s

Original: Principle (Dale, 1935).
Revised: Principle (Dale, 1935).

Original: As discussed in the last section, the combination of only positive weights and no bias term gives the added benefit of ensuring that all neuronal activations are positive, allowing for the use of the simpler k annealing algorithm.
Revised: As discussed in the last section, the combination of only positive weights and no bias term gives the added benefit of ensuring that all neuronal activations are positive, allowing for the use of the simpler k annealing algorithm.

Original: Coincidentally, having only positive weights also gives better continual learning performance as shown in the ablations of Table 2.
Revised: Coincidentally, having only positive weights also gives better continual learning performance as shown in the ablations of Table 2.

Original: A final benefit is the creation of sparse weights but this advantage has its limitations.
Revised: A final benefit is the creation of sparse weights but this advantage has its limitations.

Original: In models trained on our ConvMixer embeddings, we found only ∼ 20% weight sparsity.
Revised: In models trained on our ConvMixer embeddings, we found only ∼ 20% weight sparsity.

Original: Meanwhile, when trying to jointly train a ConvMixer with SDM on top, too many weights were set to 0, resulting in failed training runs.
Revised: Meanwhile, when trying to jointly train a ConvMixer with SDM on top, too many weights were set to 0, resulting in failed training runs.

Original: Backpropagation - Gradient descent via backpropagation used to train MLPs is likely to be biologically implausible (Lillicrap et al., 2020; Goodfellow et al., 2015).
Revised: Backpropagation - Gradient descent via backpropagation used to train MLPs is likely to be biologically implausible (Lillicrap et al., 2020; Goodfellow et al., 2015).

Original: However, there exist a number of local Hebbian learning rules associated with inhibitory interneurons and manifold tiling that we see as viable alternatives (Sengupta et al., 2018; Krotov & Hopfield, 2019; Gozel & Gerstner,
Revised: However, there exist a number of local Hebbian learning rules associated with inhibitory interneurons and manifold tiling that we see as viable alternatives (Sengupta et al., 2018; Krotov & Hopfield, 2019; Gozel & Gerstner,

Original: In addition, the cerebellum receives supervised learning signals, via climbing fibers 10 , that are known to be capable of encoding heteroassociative relationships such as an eyeblink response to a tone.
Revised: In addition, the cerebellum receives supervised learning signals, via climbing fibers 11 , that are known to be capable of encoding heteroassociative relationships such as an eyeblink response to a tone.

Original: This circuitry is considered in (Sezener et al., 2021) and presents yet another alternative to backpropagation for training the cerebellum.
Revised: This circuitry is considered in (Sezener et al., 2021) and presents yet another alternative to backpropagation for training the cerebellum.

Original: A. SDM W RITE O PERATION R ELATION TO MLP B ACKPROPAGATION
Revised: A. SDM W RITE O PERATION R ELATION TO MLP B ACKPROPAGATION

Original: The original SDM write operation directly updates the neuron value vector with the pattern value x v = x v + α p v where α weights the pattern by the amount each neuron was activated.
Revised: The original SDM write operation directly updates the neuron value vector with the pattern value x v = x v + α p v where α weights the pattern by the amount each neuron was activated.

Original: Meanwhile, the backpropgation write operation updates x v with the error between the model output and the true class one-hot (using cross entropy loss).
Revised: Meanwhile, the backpropgation write operation updates x v with the error between the model output and the true class one-hot (using cross entropy loss).

Original: During model training all inputs are considered write operations.
Revised: During model training all inputs are considered write operations.

Original: After training, during inference, all inputs are considered queries that perform the SDM read operation.
Revised: After training, during inference, all inputs are considered queries that perform the SDM read operation.

Original: p a corresponds to a CIFAR image and p v is a one hot label of its encoding.
Revised: p a corresponds to a CIFAR image and p v is a one hot label of its encoding.

Original: There are a few reasons why this difference is compatible with SDM:
Revised: There are a few reasons why this difference is compatible with SDM:

Original: First, an optimal solution for x v that will result in zero error is the one hot encoding used by the original SDM write operation.
Revised: First, an optimal solution for x v that will result in zero error is the one hot encoding used by the original SDM write operation.

Original: Second, the original SDM write is only appropriate when the neuron addresses are fixed and x v is the only thing learnt.
Revised: Second, the original SDM write is only appropriate when the neuron addresses are fixed and x v is the only thing learnt.

Original: Otherwise, as neurons update their address x a , this changes the patterns they are activated by and what p v they should store in x v .
Revised: Otherwise, as neurons update their address x a , this changes the patterns they are activated by and what p v they should store in x v .

Original: Using the backpropagation approach to continuously update x v as a function of the patterns it is currently activated by is a viable solution.
Revised: Using the backpropagation approach to continuously update x v as a function of the patterns it is currently activated by is a viable solution.

Original: Finally, from a biological perspective, the error signal used by backpropagation is a closer approximation to how the cerebellar circuit that SDM maps to updates x v (Bidirectional learning in upbound and downbound microzones of the cerebellum, De Zeeuw, 2021).
Revised: Finally, from a biological perspective, the error signal used by backpropagation is a closer approximation to how the cerebellar circuit that SDM maps to updates x v (Bidirectional learning in upbound and downbound microzones of the cerebellum, De Zeeuw, 2021).

Original: While backpropagation through multiple layers of a deep network has been argued to be biologically implausible, this update to the output layer is directly connected to the error computation making it possible.
Revised: While backpropagation through multiple layers of a deep network has been argued to be biologically implausible, this update to the output layer is directly connected to the error computation making it possible.

Original: In summary, SDM will explicitly write in p v while the MLP with backprop will compute a delta between p v and the network output but this approach is compatible with the same solution, works better when also learning neuron addresses, and is likely to be more biologically plausible.
Revised: In summary, SDM will explicitly write in p v while the MLP with backprop will compute a delta between p v and the network output but this approach is compatible with the same solution, works better when also learning neuron addresses, and is likely to be more biologically plausible.

Original: A. R ATE C ODE A CTIVATIONS M AINTAIN SDM’ S A PPROXIMATION TO T RANSFORMER A TTENTION
Revised: A. R ATE C ODE A CTIVATIONS M AINTAIN SDM’ S A PPROXIMATION TO T RANSFORMER A TTENTION

Original: It was shown in (Bricken & Pehlevan, 2021) that the Attention operation of Transformers closely approximates the SDM read operation (Vaswani et al., 2017).
Revised: It was shown in (Bricken & Pehlevan, 2021) that the Attention operation of Transformers closely approximates the SDM read operation (Vaswani et al., 2017).

Original: This is because the weighting assigned to each pattern in SDM is approximately exponential, resulting in the softmax rule.
Revised: This is because the weighting assigned to each pattern in SDM is approximately exponential, resulting in the softmax rule.

Original: However, this result is derived where SDM has binary activations of neurons when reading and writing, not weighting each pattern by its distance.
Revised: However, this result is derived where SDM has binary activations of neurons when reading and writing, not weighting each pattern by its distance.

Original: Here, we show that with linear or exponential pattern weightings, SDM remains a close approximation to Attention.
Revised: Here, we show that with linear or exponential pattern weightings, SDM remains a close approximation to Attention.

Original: This makes the results of previous work continue to hold in our case where SDM is written as an MLP, giving an interesting relation between MLPs and Attention (see Section 6).
Revised: This makes the results of previous work continue to hold in our case where SDM is written as an MLP, giving an interesting relation between MLPs and Attention (see Section 6).

Original: The algorithm derived in (Bricken & Pehlevan, 2021) for the size of each circle intersection is summarized before showing how it can implement the weighting coefficient and the effects of this weighting.
Revised: The algorithm derived in (Bricken & Pehlevan, 2021) for the size of each circle intersection is summarized before showing how it can implement the weighting coefficient and the effects of this weighting.

Original: As in the original SDM formulation, we are using n dimensional binary vectors with a Hamming activation threshold of d and the distance between the addresses of a query ξ and pattern p a is d v :=
Revised: As in the original SDM formulation, we are using n dimensional binary vectors with a Hamming activation threshold of d and the distance between the addresses of a query ξ and pattern p a is d v :=

Original: d ( ξ , p a ) .
Revised: d ( ξ , p a ) .

Original: We can group the elements of ξ and p a into two disjoint sets: the n − d v elements where they agree and the d v elements where they disagree 11 :
Revised: We can group the elements of ξ and p a into two disjoint sets: the n − d v elements where they agree and the d v elements where they disagree 12 :

Original: [Equation]
Revised: [Equation]

Original: Now imagine a third vector, representing a potential neuron address.
Revised: Now imagine a third vector, representing a potential neuron address.

Original: This neuron has four possible groups that the elements of its vector can fall into:
Revised: This neuron has four possible groups that the elements of its vector can fall into:

Original: • a - agree with both ξ and p a • z - disagree with both ξ and p a • b - agree with ξ and disagree with p a • c - agree with p a and disagree with ξ
Revised: • a - agree with both ξ and p a • z - disagree with both ξ and p a • b - agree with ξ and disagree with p a • c - agree with p a and disagree with ξ

Original: We want to constraint the values of a , z , b and c such that the neuron address exists inside the circle intersection between ξ and p a .
Revised: We want to constraint the values of a , z , b and c such that the neuron address exists inside the circle intersection between ξ and p a .

Original: This produces the following constraints:
Revised: This produces the following constraints:

Original: [Equation]
Revised: [Equation]

Original: Using the notation of (Vaswani et al., 2017), we can write the total number of neurons that exist in the intersection of the read and write circles as:
Revised: Using the notation of (Vaswani et al., 2017), we can write the total number of neurons that exist in the intersection of the read and write circles as:

Original: [Equation]
Revised: [Equation]

Original: , where introduce the weight coefficient w Type () that can be:
Revised: , where introduce the weight coefficient w Type () that can be:

Original: [Equation]
Revised: [Equation]

Original: where w Binary is the original weighting of 1 for everything; w Linear applies a linear decay fromfor a perfect vector match to 0 for the maximum distance allowed between vectors; and w Exp is an exponential decay weighting that uses a β coefficient for its decay rate.
Revised: where w Binary is the original weighting of 1 for everything; w Linear applies a linear decay fromfor a perfect vector match to 0 for the maximum distance allowed between vectors; and w Exp is an exponential decay weighting that uses a β coefficient for its decay rate.

Original: As we will now show, the linear weighting applies the most weight to patterns right in the middle of the pattern and query with a gradual, symmetric decline around this point.
Revised: As we will now show, the linear weighting applies the most weight to patterns right in the middle of the pattern and query with a gradual, symmetric decline around this point.

Original: Meanwhile, the exponential weighting cancels out to apply a constant weight re-scaling to everything.
Revised: Meanwhile, the exponential weighting cancels out to apply a constant weight re-scaling to everything.

Original: Letting x represent the distance of a neuron to the read and write vectors it is weighted by, x := ( n − z ) − ( a + c ) =
Revised: Letting x represent the distance of a neuron to the read and write vectors it is weighted by, x := ( n − z ) − ( a + c ) =

Original: b − z .
Revised: b − z .

Original: Because we want to know the weighting coefficient that applies to neurons at all possible distances from the pattern and query, without loss of generality we can set z = 0 for the analysis that follows:
Revised: Because we want to know the weighting coefficient that applies to neurons at all possible distances from the pattern and query, without loss of generality we can set z = 0 for the analysis that follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C =
Revised: where C =

Original: ( d v + z ) is a constant as is Z = n 2 − n C n 2 .
Revised: ( d v + z ) is a constant as is Z = n 2 − n C n 2 .

Original: We can take the first and second derivatives of Eq.
Revised: We can take the first and second derivatives of Eq.

Original: 10 to know that the most weight is applied at the distance right between the read and write vectors,
Revised: 10 to know that the most weight is applied at the distance right between the read and write vectors,

Original: C / 2 .
Revised: C / 2 .

Original: This linear weighting applies different weights to neurons at different distances.
Revised: This linear weighting applies different weights to neurons at different distances.

Original: As a result, the patterns stored in superposition will also have different weightings.
Revised: As a result, the patterns stored in superposition will also have different weightings.

Original: However, empirically, this has a negligible effect on the SDM exponential approximation, as shown in Fig.
Revised: However, empirically, this has a negligible effect on the SDM exponential approximation, as shown in Fig.

Original: We hypothesize that this is due to two factors: (i) the difference in weight values is not particularly large; (ii) neurons receiving the largest weights are the most numerous.
Revised: We hypothesize that this is due to two factors: (i) the difference in weight values is not particularly large; (ii) neurons receiving the largest weights are the most numerous.

Original: Therefore, the weighting is correlated with the approximately exponential decay in the number of neurons that exist in the circle intersection as vector distance changes.
Revised: Therefore, the weighting is correlated with the approximately exponential decay in the number of neurons that exist in the circle intersection as vector distance changes.

Original: As for the exponential, it cancels to a constant term that depends upon the choice of β :
Revised: As for the exponential, it cancels to a constant term that depends upon the choice of β :

Original: [Equation]
Revised: [Equation]

Original: This constant term modification to the weighting of all patterns is then removed by the normalization term in the softmax operation, resulting in no effect on the output.
Revised: This constant term modification to the weighting of all patterns is then removed by the normalization term in the softmax operation, resulting in no effect on the output.

Original: We confirm our results empirically for the three optimal SDM hamming distances
Revised: We confirm our results empirically for the three optimal SDM hamming distances

Original: d ∗ and n =dimensional
Revised: d ∗ and n =dimensional

Original: canonical Transformer Attention setting used throughout (Bricken & Pehlevan, 2021) in Fig.
Revised: canonical Transformer Attention setting used throughout (Bricken & Pehlevan, 2021) in Fig.

Original: [Figure]
Revised: [Figure]

Original: B GABA S WITCH B. B IOLOGICALLY I MPLAUSIBLE S OLUTIONS TO THE D EAD N EURON P ROBLEM
Revised: B GABA S WITCH B. B IOLOGICALLY I MPLAUSIBLE S OLUTIONS TO THE D EAD N EURON P ROBLEM

Original: The dead neuron problem is where neurons are never activated by any input data and are a waste of resources.
Revised: The dead neuron problem is where neurons are never activated by any input data and are a waste of resources.

Original: This is particularly a problem with the Top-K activation function, especially at initialization where it is possible for a small subset of neurons that happen to already exist closest to the data manifold to always be activated and in the Top-K. This means that none of the other neurons are ever activated, and thus never receive weight updates, preventing them from learning the data manifold and becoming useful.
Revised: This is particularly a problem with the Top-K activation function, especially at initialization where it is possible for a small subset of neurons that happen to already exist closest to the data manifold to always be activated and in the Top-K. This means that none of the other neurons are ever activated, and thus never receive weight updates, preventing them from learning the data manifold and becoming useful.

Original: The first set of strategies to solve the dead neuron problem initializes neurons immediately on or near the data manifold, for example, using PCA or SVD (Rumelhart & Zipser, 1985; McInnes &
Revised: The first set of strategies to solve the dead neuron problem initializes neurons immediately on or near the data manifold, for example, using PCA or SVD (Rumelhart & Zipser, 1985; McInnes &

Original: Healy, 2018).
Revised: Healy, 2018).

Original: Biologically, this would assume that priors on the location of the data manifold are encoded in our genome and pre-determine the structure of neuronal dendrites.
Revised: Biologically, this would assume that priors on the location of the data manifold are encoded in our genome and pre-determine the structure of neuronal dendrites.

Original: However, dendrites often form in a highly dynamic and stochastic way (Dhar et al., 2018; Purves et al., 2001).
Revised: However, dendrites often form in a highly dynamic and stochastic way (Dhar et al., 2018; Purves et al., 2001).

Original: In addition, this solution requires the genome to store a large amount of information and would limit an organism’s adaptability to learn new data manifolds during its development, for example by moving to a new environmental niche.
Revised: In addition, this solution requires the genome to store a large amount of information and would limit an organism’s adaptability to learn new data manifolds during its development, for example by moving to a new environmental niche.

Original: It is popular in deep learning to initialize weights not using the data directly but with optimized strategies such as Xavier (Glorot & Bengio, 2010) that empirically work well, even for the sparse ReLU activation function.
Revised: It is popular in deep learning to initialize weights not using the data directly but with optimized strategies such as Xavier (Glorot & Bengio, 2010) that empirically work well, even for the sparse ReLU activation function.

Original: However, as evidenced by the continued development of new initialization strategies, this approach remains heuristic and does not generalize across datasets, networks, and training techniques (Aguirre, 2019;
Revised: However, as evidenced by the continued development of new initialization strategies, this approach remains heuristic and does not generalize across datasets, networks, and training techniques (Aguirre, 2019;

Original: Chang et al., 2020b;
Revised: Chang et al., 2020b;

Original: Goodfellow et al., 2015).
Revised: Goodfellow et al., 2015).

Original: The second set of approaches to solve the dead neuron problem ensures that all neurons, independent of their initialization, are active and update to learn the data manifold.
Revised: The second set of approaches to solve the dead neuron problem ensures that all neurons, independent of their initialization, are active and update to learn the data manifold.

Original: This is a major cited reason why ReLU is often avoided (Goodfellow et al., 2015; Hendrycks & Gimpel, 2016), even though in practice it often does not seem to significantly harm performance (Pedamonti, 2018; Trockman & Kolter, 2022).
Revised: This is a major cited reason why ReLU is often avoided (Goodfellow et al., 2015; Hendrycks & Gimpel, 2016), even though in practice it often does not seem to significantly harm performance (Pedamonti, 2018; Trockman & Kolter, 2022).

Original: The Top-K model of (Ahmad & Scheinkman, 2019), VQ-VAE (van den Oord et al., 2017), and Switch Transformer (Fedus et al., 2021) all address their dead neuron problems in biologically implausible ways.
Revised: The Top-K model of (Ahmad & Scheinkman, 2019), VQ-VAE (van den Oord et al., 2017), and Switch Transformer (Fedus et al., 2021) all address their dead neuron problems in biologically implausible ways.

Original: When using the Top-K activation function, the work of (Ahmad & Scheinkman, 2019) artificially “boosted” the ranking for each neuron to be in the Top-K, with more inactive neurons getting higher rankings.
Revised: When using the Top-K activation function, the work of (Ahmad & Scheinkman, 2019) artificially “boosted” the ranking for each neuron to be in the Top-K, with more inactive neurons getting higher rankings.

Original: This solution makes the biologically implausible assumption that neurons fire as a function of how inactive they are and undergo ubiquitous antiHebbian plasticity.
Revised: This solution makes the biologically implausible assumption that neurons fire as a function of how inactive they are and undergo ubiquitous antiHebbian plasticity.

Original: Meanwhile, the VQ-VAE and Switch Transformer both used terms in their loss function to increase the utilization of “dead” code vectors and expert modules, respectively.
Revised: Meanwhile, the VQ-VAE and Switch Transformer both used terms in their loss function to increase the utilization of “dead” code vectors and expert modules, respectively.

Original: B. GABA S WITCH B IOLOGY
Revised: B. GABA S WITCH B IOLOGY

Original: We summarize how the GABA switch works biologically in Fig.
Revised: We summarize how the GABA switch works biologically in Fig.

Original: 8 (Gozel & Gerstner, 2021).
Revised: 8 (Gozel & Gerstner, 2021).

Original: Neurons are excited by GABA early in development before being inhibited by it due to changes in intracellular CL − concentration.
Revised: Neurons are excited by GABA early in development before being inhibited by it due to changes in intracellular CL − concentration.

Original: After neurogenesis, neurons express NKCC1 which imports CL − into the cell causing the CL − reversal potential to be more positive than resting potential (Gozel & Gerstner, 2021; Heigele et al., 2016).
Revised: After neurogenesis, neurons express NKCC1 which imports CL − into the cell causing the CL − reversal potential to be more positive than resting potential (Gozel & Gerstner, 2021; Heigele et al., 2016).

Original: When GABA is present, CL − flows towards its reversal potential, resulting in depolarization.
Revised: When GABA is present, CL − flows towards its reversal potential, resulting in depolarization.

Original: Over time, this GABA activation indirectly results in increased
Revised: Over time, this GABA activation indirectly results in increased

Original: KCC2, which instead pumps CL − out of the cell (Ganguly et al., 2001; Connor et al., 1987).
Revised: KCC2, which instead pumps CL − out of the cell (Ganguly et al., 2001; Connor et al., 1987).

Original: This makes the CL − reversal potential more negative than the resting potential and hyper-polarization during GABA activation.
Revised: This makes the CL − reversal potential more negative than the resting potential and hyper-polarization during GABA activation.

Original: This biological mechanism suggests that our GABA switch implementation in Eq.
Revised: This biological mechanism suggests that our GABA switch implementation in Eq.

Original: 12 could be made more sophisticated.
Revised: 12 could be made more sophisticated.

Original: Rather than counting up the number of times the neuron is simply active (a binary outcome), it could switch as a function of the actual activation amount that the neuron experiences or size of gradient updates.
Revised: Rather than counting up the number of times the neuron is simply active (a binary outcome), it could switch as a function of the actual activation amount that the neuron experiences or size of gradient updates.

Original: B. GABA S WITCH P RACTICAL C ONSIDERATIONS
Revised: B. GABA S WITCH P RACTICAL C ONSIDERATIONS

Original: There are three conclusions from this section on when and how to use the GABA switch to avoid dead neurons, improving continual learning:
Revised: There are three conclusions from this section on when and how to use the GABA switch to avoid dead neurons, improving continual learning:

Original: • If the positive weight constraint is being used then annealing k gives the same performance as the full GABA switch.
Revised: • If the positive weight constraint is being used then annealing k gives the same performance as the full GABA switch.

Original: If you allow for negative weights then the GABA switch should be used.
Revised: If you allow for negative weights then the GABA switch should be used.

Original: • The subtraction operation works much better for continual learning than masking.
Revised: • The subtraction operation works much better for continual learning than masking.

Original: However, the inverse is true in a non continual learning setting.
Revised: However, the inverse is true in a non continual learning setting.

Original: We provide some initial analysis of why this is the case.
Revised: We provide some initial analysis of why this is the case.

Original: • The value of s for the GABA switch should consider the complexity of the data manifold, optimizer, and learning rate.
Revised: • The value of s for the GABA switch should consider the complexity of the data manifold, optimizer, and learning rate.

Original: When in doubt there is no harm in setting s larger aside from requiring more training epochs.
Revised: When in doubt there is no harm in setting s larger aside from requiring more training epochs.

Original: We first give the full GABA switch algorithm that was developed and is used for the SDMLP throughout the paper.
Revised: We first give the full GABA switch algorithm that was developed and is used for the SDMLP throughout the paper.

Original: We then outline why the k annealing approximation works just as well when we constrain our weights and inputs to be only positive.
Revised: We then outline why the k annealing approximation works just as well when we constrain our weights and inputs to be only positive.

Original: Next, we discuss why subtracting to enforce
Revised: Next, we discuss why subtracting to enforce

Original: Top-K instead of masking leads to better continual learning performance.
Revised: Top-K instead of masking leads to better continual learning performance.

Original: We conclude with other considerations on how to avoid dead neurons with interactions between the data manifold, learning rate, and hyperparameter s .
Revised: We conclude with other considerations on how to avoid dead neurons with interactions between the data manifold, learning rate, and hyperparameter s .

Original: The GABA Switch Algorithm
Revised: The GABA Switch Algorithm

Original: Formally, the GABA switch algorithm is implemented as:
Revised: Formally, the GABA switch algorithm is implemented as:

Original: [Equation]
Revised: [Equation]

Original: This algorithm is slightly more complex than the K annealing one presented in Eq.
Revised: This algorithm is slightly more complex than the K annealing one presented in Eq.

Original: 4 of the main text.
Revised: 4 of the main text.

Original: C i is a counter for each neuron, recording the number of times that it has been activated since the start of training.
Revised: C i is a counter for each neuron, recording the number of times that it has been activated since the start of training.

Original: λ i linearly increases from -1 to 1 as a function of C i , and [ · ] + is the ReLU operation.
Revised: λ i linearly increases from -1 to 1 as a function of C i , and [ · ] + is the ReLU operation.

Original: s is a hyperparameter that determines the number of activations required for this neuron to switch from being excitated by GABA to being inhibited.
Revised: s is a hyperparameter that determines the number of activations required for this neuron to switch from being excitated by GABA to being inhibited.

Original: When training on top of the ConvMixer embeddings, we found that s = 250 , 000 ensured all neurons could move onto the data manifold, resulting in none being dead.
Revised: When training on top of the ConvMixer embeddings, we found that s = 250 , 000 ensured all neurons could move onto the data manifold, resulting in none being dead.

Original: As an example, if we assume all neurons are activated before the
Revised: As an example, if we assume all neurons are activated before the

Original: GABA switch, with this s value the switch occurs after just 2.5 epochs of training on CIFAR (there are 50,000 training examples).
Revised: GABA switch, with this s value the switch occurs after just 2.5 epochs of training on CIFAR (there are 50,000 training examples).

Original: However, this value depends significantly upon the learning rate and complexity of the manifold where in the case of training on raw CIFAR10 images, to ensure there are no dead neurons we set s = 5 , 000 , 000 requiring 100 epochs for the GABA switch to occur.
Revised: However, this value depends significantly upon the learning rate and complexity of the manifold where in the case of training on raw CIFAR10 images, to ensure there are no dead neurons we set s = 5 , 000 , 000 requiring 100 epochs for the GABA switch to occur.

Original: However, in this case we did not try to find the lower bound on s .
Revised: However, in this case we did not try to find the lower bound on s .

Original: Why k annealing works just as well as the GABA switch when weights are positive
Revised: Why k annealing works just as well as the GABA switch when weights are positive

Original: Because of Dale’s Principle, we implement SDM with positive weights and inputs.
Revised: Because of Dale’s Principle, we implement SDM with positive weights and inputs.

Original: Empirically, this also boosts continual learning performance and introduces weight sparsity 12 that would increase computational efficiency if unstructured sparsity can be taken advantage of (Wang, 2020).
Revised: Empirically, this also boosts continual learning performance and introduces weight sparsity 13 that would increase computational efficiency if unstructured sparsity can be taken advantage of (Wang, 2020).

Original: When positive weights are used, we found no change in the number of dead neurons or continual learning performance using Eq.
Revised: When positive weights are used, we found no change in the number of dead neurons or continual learning performance using Eq.

Original: 4 linear annealing.
Revised: 4 linear annealing.

Original: This was true for both the ImageNet32 ConvMixer embeddings and raw pixel datasets.
Revised: This was true for both the ImageNet32 ConvMixer embeddings and raw pixel datasets.

Original: However, when negative weights are allowed this was no longer the case and only the full GABA switch algorithm avoided dead neurons.
Revised: However, when negative weights are allowed this was no longer the case and only the full GABA switch algorithm avoided dead neurons.

Original: We speculate that this is because when GABA is excitatory, the full algorithm will inject positive activation into neurons that may otherwise have negative activity and not fire thus failing to get a gradient update.
Revised: We speculate that this is because when GABA is excitatory, the full algorithm will inject positive activation into neurons that may otherwise have negative activity and not fire thus failing to get a gradient update.

Original: Meanwhile, with positive weights and inputs the activation for every neuron is always positive resulting in every neuron being active unless Top-K is enforced.
Revised: Meanwhile, with positive weights and inputs the activation for every neuron is always positive resulting in every neuron being active unless Top-K is enforced.

Original: Thus the positive weights and inputs ensure that every neuron receives gradient updates and moves onto the data manifold.
Revised: Thus the positive weights and inputs ensure that every neuron receives gradient updates and moves onto the data manifold.

Original: In Figure 9, we present this result training directly on CIFAR10 pixels, a more difficult manifold to learn than the ConvMixer embeddings.
Revised: In Fig. 9, we present this result training directly on CIFAR10 pixels, a more difficult manifold to learn than the ConvMixer embeddings.

Original: Note that while positive weights allow for linear k annealing, it can also result in all weights be set to 0, leading to dead neurons and model training to fail.
Revised: Note that while positive weights allow for linear k annealing, it can also result in all weights be set to 0, leading to dead neurons and model training to fail.

Original: Why subtracting works better than masking for Continual Learning
Revised: Why subtracting works better than masking for Continual Learning

Original: Empirically, masking results in much higher validation accuracies during pretraining than subtraction (Fig. 10b).
Revised: Empirically, masking results in much higher validation accuracies during pretraining than subtraction (Fig. 10b).

Original: However, during continual learning, subtraction works better as shown in the ablations of Table 2 and Fig. 10a.
Revised: However, during continual learning, subtraction works better as shown in the ablations of Table 2 and Fig. 10a.

Original: To work out why subtracting is better for continual learning we analyzed the learning dynamics of both in Appendix H and discuss them here.
Revised: To work out why subtracting is better for continual learning we analyzed the learning dynamics of both in App. H and discuss them here.

Original: We leave an analysis of why masking works better during pretraining to future work as we do not care about maximizing validation accuracy for our pretraining task here.
Revised: We leave an analysis of why masking works better during pretraining to future work as we do not care about maximizing validation accuracy for our pretraining task here.

Original: Top-K with subtraction utilizes more neurons during continual learning as shown in Fig. 10c.
Revised: Top-K with subtraction utilizes more neurons during continual learning as shown in Fig. 10c.

Original: This result is supported by Appendix H Fig.
Revised: This result is supported by App. H Fig.

Original: 23 where there are fewer neurons that are consistently activated and they are more polysemantic, being used for multiple tasks.
Revised: 23 where there are fewer neurons that are consistently activated and they are more polysemantic, being used for multiple tasks.

Original: Motivated by the stale momentum results of Appendix D, we wondered if the larger activation values preserved by masking may lead to more dead neurons when using SGDM as our momentum based optimizer.
Revised: Motivated by the stale momentum results of App. D, we wondered if the larger activation values preserved by masking may lead to more dead neurons when using SGDM as our momentum based optimizer.

Original: However, using SGD did not lead to any change in results and in hindsight our gradients being less than 1 means that the results in Appendix D hold independent of the gradient values.
Revised: However, using SGD did not lead to any change in results and in hindsight our gradients being less than 1 means that the results in App. D hold independent of the gradient values.

Original: This means that the gradients must actually be smaller for subtraction and lead to slower updating.
Revised: This means that the gradients must actually be smaller for subtraction and lead to slower updating.

Original: At first glance, the subtraction operation just reduces the activity of the k neurons that remain active by a constant term.
Revised: At first glance, the subtraction operation just reduces the activity of the k neurons that remain active by a constant term.

Original: This will scale the size of each gradient, equivalent to modifying the learning rate.
Revised: This will scale the size of each gradient, equivalent to modifying the learning rate.

Original: However, the situation is more complex.
Revised: However, the situation is more complex.

Original: The activity of the k neurons and their learning rate is conditioned upon the activity of the k + 1 th neuron.
Revised: The activity of the k neurons and their learning rate is conditioned upon the activity of the k + 1 th neuron.

Original: If this neuron is almost as active as those in the k subset, all of their activities will be very close to zero.
Revised: If this neuron is almost as active as those in the k subset, all of their activities will be very close to zero.

Original: Meanwhile, if the k + 1 th neuron is much less active than the k subset, their activities will remain large and they will receive a larger gradient update.
Revised: Meanwhile, if the k + 1 th neuron is much less active than the k subset, their activities will remain large and they will receive a larger gradient update.

Original: These two situations correspond to the input being in densely and sparsely tiled regions of the data manifold, respectively.
Revised: These two situations correspond to the input being in densely and sparsely tiled regions of the data manifold, respectively.

Original: We can think of this situation as approximating a Bayesian one where we have a mixture of Gaussians defining a posterior distribution over the data manifold.
Revised: We can think of this situation as approximating a Bayesian one where we have a mixture of Gaussians defining a posterior distribution over the data manifold.

Original: The number of neurons in a region is inversely proportional to the variance in the distribution at that location and thus changes the amount the likelihood of our current data point updates the distribution.
Revised: The number of neurons in a region is inversely proportional to the variance in the distribution at that location and thus changes the amount the likelihood of our current data point updates the distribution.

Original: In other words, the subtraction operation introduces a dynamic learning rate.
Revised: In other words, the subtraction operation introduces a dynamic learning rate.

Original: [Figure]
Revised: [Figure]

Original: During continual learning when the task changes and a new subregion of the data manifold is presented, we want those neurons closest to the subregion and only those neurons to update and move onto it.
Revised: During continual learning when the task changes and a new subregion of the data manifold is presented, we want those neurons closest to the subregion and only those neurons to update and move onto it.

Original: In our stochastic optimization setting, it is possible that neurons not necessarily closest to new subregion are updated towards it instead, leading to an over-allocation of neurons to the new task.
Revised: In our stochastic optimization setting, it is possible that neurons not necessarily closest to new subregion are updated towards it instead, leading to an over-allocation of neurons to the new task.

Original: We hypothesize that the subtraction operation helps avoid this problem by making the learning rate both lower and dynamic.
Revised: We hypothesize that the subtraction operation helps avoid this problem by making the learning rate both lower and dynamic.

Original: However, we believe the dynamic learning effect is more influential than stochasticity because even when increasing the batch size from 128 to 10,000, we see that subtraction is still more robust to catastrophic forgetting.
Revised: However, we believe the dynamic learning effect is more influential than stochasticity because even when increasing the batch size from 128 to 10,000, we see that subtraction is still more robust to catastrophic forgetting.

Original: It is likely that the reduced and dynamic learning rate is a doubled edged sword where while it is better at remembering, it is also worse at learning new information.
Revised: It is likely that the reduced and dynamic learning rate is a doubled edged sword where while it is better at remembering, it is also worse at learning new information.

Original: Both in pretraining and within each data split, the subtraction achieves worse training and validation accuracies.
Revised: Both in pretraining and within each data split, the subtraction achieves worse training and validation accuracies.

Original: Why the subtraction operation does worse during the pretraining on a single task we leave to future work.
Revised: Why the subtraction operation does worse during the pretraining on a single task we leave to future work.

Original: As an interesting aside, the Top-K subtraction operation implemented by an inhibitory interneuron when k = 1 is the same as a Vickrey second price auction used in (Chang et al., 2020a) to model neurons as individual agents with their own utility functions.
Revised: As an interesting aside, the Top-K subtraction operation implemented by an inhibitory interneuron when k = 1 is the same as a Vickrey second price auction used in (Chang et al., 2020a) to model neurons as individual agents with their own utility functions.

Original: The winning neuron places a “bid” represented by its activity amount.
Revised: The winning neuron places a “bid” represented by its activity amount.

Original: It has its activity subtracted by that of the second most active neuron (thus paying the second highest price) and its remaining activity determines its exposure to the next gradient update.
Revised: It has its activity subtracted by that of the second most active neuron (thus paying the second highest price) and its remaining activity determines its exposure to the next gradient update.

Original: Other GABA Switch Considerations - Data Manifolds , Optimizers
Revised: Other GABA Switch Considerations - Data Manifolds, Optimizers,

Original: and Learning Rates
Revised: and Learning Rates

Original: Note that for the ConvMixer embedding, dead neurons are not as much of a problem and either the GABA switch or k annealing can avoid any dead neurons within just a few epochs and with or without positive weights.
Revised: Note that for the ConvMixer embedding, dead neurons are not as much of a problem and either the GABA switch or k annealing can avoid any dead neurons within just a few epochs and with or without positive weights.

Original: This is because the manifold is much easier to learn as it is lower dimensional both in the size of vectors (256 versus 3072) and the data manifold being far more meaningfully correlated.
Revised: This is because the manifold is much easier to learn as it is lower dimensional both in the size of vectors (256 versus 3072) and the data manifold being far more meaningfully correlated.

Original: Fig.
Revised: Fig.

Original: 11 shows how a learning rate that is too low will cause SGD to produce dead neurons.
Revised: 11 shows how a learning rate that is too low will cause SGD to produce dead neurons.

Original: The number of neurons dying being inversely proportional to the learning rate and the sudden jump in dead neurons before it plateaus just after the GABA switch both support the theory that the learning rate is too low.
Revised: The number of neurons dying being inversely proportional to the learning rate and the sudden jump in dead neurons before it plateaus just after the GABA switch both support the theory that the learning rate is too low.

Original: This means that neurons do not update quickly enough onto the data manifold before the GABA switch occurs.
Revised: This means that neurons do not update quickly enough onto the data manifold before the GABA switch occurs.

Original: This sudden jump and plateau in dead neurons is in contrast to Adam and RMSProp optimizers shown in Fig.
Revised: This sudden jump and plateau in dead neurons is in contrast to Adam and RMSProp optimizers shown in Fig.

Original: 15 where neurons will continue dying over time due to the stale momentum problems (Kingma & Ba, 2015; Geoffrey Hinton, 2012).
Revised: 15 where neurons will continue dying over time due to the stale momentum problems (Kingma & Ba, 2015; Geoffrey Hinton, 2012).

Original: Meanwhile, SGDM does not suffer from Stale Momentum problems to the same extent and is much more robust to choice of learning rate making it a good choice for pretraining.
Revised: Meanwhile, SGDM does not suffer from Stale Momentum problems to the same extent and is much more robust to choice of learning rate making it a good choice for pretraining.

Original: However, it is still suffers somewhat from the stale momentum problem in the continual learning setting as shown in the ablations of Table 2.
Revised: However, it is still suffers somewhat from the stale momentum problem in the continual learning setting as shown in the ablations of Table 2.

Original: To avoid dead neurons when using SGD either the learning rate can be tuned, the GABA switch value s can be increased, or a sparse optimizer that avoids the stale momentum problem can be used (however, the success of these sparse optimizers remains to be validated in future work).
Revised: To avoid dead neurons when using SGD either the learning rate can be tuned, the GABA switch value s can be increased, or a sparse optimizer that avoids the stale momentum problem can be used (however, the success of these sparse optimizers remains to be validated in future work).

Original: Also keep in mind that the learning rate chosen will be affected by the L 2 normalization.
Revised: Also keep in mind that the learning rate chosen will be affected by the L 2 normalization.

Original: C T OP -K C. I NHIBITORY I NTERNEURONS A PPROXIMATE T OP -K
Revised: C T OP -K C. I NHIBITORY I NTERNEURONS A PPROXIMATE T OP -K

Original: At a high level, inhibitory interneurons exist to regulate the firing of excitatory neurons, introducing sparse firing and keeping only the most active on.[-
-]
Revised: At a high level, inhibitory interneurons exist to regulate the firing of excitatory neurons, introducing sparse firing and keeping only the most active on.[-
-]

Original: This makes the brain likely to be highly sparse in the number of neurons firing at any given time.
Revised: This makes the brain likely to be highly sparse in the number of neurons firing at any given time.

Original: Estimates suggest that “an average neuron in the human brain transmits a spike about 0.1-2 times per second.”
Revised: Estimates suggest that “an average neuron in the human brain transmits a spike about 0.1-2 times per second.”

Original: (Impacts, 2022), while action potentials and refractory periods happen on a time interval of roughly 10ms (Sterling & Laughlin, 2015).
Revised: (Impacts, 2022), while action potentials and refractory periods happen on a time interval of roughly 10ms (Sterling & Laughlin, 2015).

Original: Assuming neurons fire randomly within this 10ms time interval, this gives a back of the envelope calculation that a maximum of 10ms/100ms = 10% neurons will fire within the time interval.
Revised: Assuming neurons fire randomly within this 10ms time interval, this gives a back of the envelope calculation that a maximum of 10ms/100ms = 10% neurons will fire within the time interval.

Original: This aligns well with the prediction of 15% by (Attwell & Laughlin, 2001).
Revised: This aligns well with the prediction of 15% by (Attwell & Laughlin, 2001).

Original: Sparse firing also makes sense from the perspective of metabolic costs where action potentials are expensive, consuming
Revised: Sparse firing also makes sense from the perspective of metabolic costs where action potentials are expensive, consuming

Original: ∼ 20% of the brain’s energy (Sterling & Laughlin, 2015; Attwell & Laughlin, 2001; Sengupta et al., 2010).
Revised: 2015; Attwell & Laughlin, 2001; Sengupta et al., 2010).

Original: However, in practice, it is unrealistic to assume that an inhibitory neuron can keep exactly k neurons active for any given input because its afferent and efferent synaptic connections are heterogeneous.
Revised: However, in practice, it is unrealistic to assume that an inhibitory neuron can keep exactly k neurons active for any given input because its afferent and efferent synaptic connections are heterogeneous.

Original: The inhibitory interneuron sums together the activations from all neurons, weighted by their presynaptic strengths and then outputs an inhibitory value that is scaled by the strengths of post synaptic connections.
Revised: The inhibitory interneuron sums together the activations from all neurons, weighted by their presynaptic strengths and then outputs an inhibitory value that is scaled by the strengths of post synaptic connections.

Original: The weighted summation of inputs to the interneuron removes information on how many neurons are firing and at what rate.
Revised: The weighted summation of inputs to the interneuron removes information on how many neurons are firing and at what rate.

Original: The heterogenous post synaptic connections will weight the effects of inhibitory on each neuron differently.
Revised: The heterogenous post synaptic connections will weight the effects of inhibitory on each neuron differently.

Original: Keeping these concerns in mind, there are a few reasons that we draw from for optimism that Top-K can be approximated.
Revised: Keeping these concerns in mind, there are a few reasons that we draw from for optimism that Top-K can be approximated.

Original: First, inhibitory horizontal cells in the retina compute the mean activation value of thousands of nearby cone photoreceptors and inhbits them so that they encode contrast (Chapter 13 of (Sterling & Laughlin, 2015)).
Revised: First, inhibitory horizontal cells in the retina compute the mean activation value of thousands of nearby cone photoreceptors and inhbits them so that they encode contrast (Chapter 13 of (Sterling & Laughlin, 2015)).

Original: Not only do horizontal cells have carefully tuned weights for their gap junction connections to compute this mean, but also will dynamically change its pre and post synaptic connectivity strengths to accurately compute mean activity under different lighting conditions.
Revised: Not only do horizontal cells have carefully tuned weights for their gap junction connections to compute this mean, but also will dynamically change its pre and post synaptic connectivity strengths to accurately compute mean activity under different lighting conditions.

Original: In a low light setting, the horizontal cell will average over a larger number of neurons to reduce the variance in its estimate of the mean (page 252 of (Sterling & Laughlin, 2015)).
Revised: In a low light setting, the horizontal cell will average over a larger number of neurons to reduce the variance in its estimate of the mean (page 252 of (Sterling & Laughlin, 2015)).

Original: If inhibitory interneurons in general come close to the level of sophistication shown by horizontal cells in calibrating their synaptic connectivity, then it is possible the interneuron can compute how much it should inhibit to implement Top-K reliably.
Revised: If inhibitory interneurons in general come close to the level of sophistication shown by horizontal cells in calibrating their synaptic connectivity, then it is possible the interneuron can compute how much it should inhibit to implement Top-K reliably.

Original: Second, looking at biological evidence of Top-K from cerebellum-like structures, there is strong evidence of Top-K being approximated in the Drosophila Mushroom Body by its Golgi interneuron analog (Lin et al., 2014).
Revised: Second, looking at biological evidence of Top-K from cerebellum-like structures, there is strong evidence of Top-K being approximated in the Drosophila Mushroom Body by its Golgi interneuron analog (Lin et al., 2014).

Original: In mammals, the evidence is more complicated with some papers finding dense granule cell activations (Giovannucci et al., 2017).
Revised: In mammals, the evidence is more complicated with some papers finding dense granule cell activations (Giovannucci et al., 2017).

Original: However, other experimental evidence and theory suggests that granule cell sparsity in the mammalian cerebellum may be a function of task complexity (Lanore et al., 2021; Xie et al., 2022).
Revised: However, other experimental evidence and theory suggests that granule cell sparsity in the mammalian cerebellum may be a function of task complexity (Lanore et al., 2021; Xie et al., 2022).

Original: The lower complexity the task, the more dense the representations can afford to be as there are fewer stimuli that must be unique encoded.
Revised: The lower complexity the task, the more dense the representations can afford to be as there are fewer stimuli that must be unique encoded.

Original: Meanwhile, high complexity tasks require a larger number of orthogonal codes resulting in the need for sparser representations that lose fidelity as a result (Xie et al., 2022).
Revised: Meanwhile, high complexity tasks require a larger number of orthogonal codes resulting in the need for sparser representations that lose fidelity as a result (Xie et al., 2022).

Original: Ultimately, the density of granule cell firing remains an open question in need of lower latency voltage indicators and the ability to record from more neurons simultaneously across a more diverse range of tasks (Lanore et al., 2021).
Revised: Ultimately, the density of granule cell firing remains an open question in need of lower latency voltage indicators and the ability to record from more neurons simultaneously across a more diverse range of tasks (Lanore et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: Finally, we tried to model the ability for an inhibitory interneuron to approximate Top-K under the simplifying assumption that its pre and post synaptic weights are fixed and homogenous.
Revised: Finally, we tried to model the ability for an inhibitory interneuron to approximate Top-K under the simplifying assumption that its pre and post synaptic weights are fixed and homogenous.

Original: We first analyzed the k neurons still on using the Hebbian learning dynamics of (Gozel & Gerstner, 2021).
Revised: We first analyzed the k neurons still on using the Hebbian learning dynamics of (Gozel & Gerstner, 2021).

Original: We then looked at our SDMLP during learning and the relationship between the total sum of excitatory activations and the k + 1 -th inhibitory value used in Eq.
Revised: We then looked at our SDMLP during learning and the relationship between the total sum of excitatory activations and the k + 1 -th inhibitory value used in Eq.

Original: 4 to keep only k neurons still on.
Revised: 4 to keep only k neurons still on.

Original: The work that introduces the GABA switch (Gozel & Gerstner, 2021) uses a Hebbian learning rule with an inhibitory interneuron.
Revised: The work that introduces the GABA switch (Gozel & Gerstner, 2021) uses a Hebbian learning rule with an inhibitory interneuron.

Original: An MNIST digit is presented for the network to learn and it obeys the following ordinary differential equations to determine the activities of the excitatory and inhibitory neurons:
Revised: An MNIST digit is presented for the network to learn and it obeys the following ordinary differential equations to determine the activities of the excitatory and inhibitory neurons:

Original: [Equation]
Revised: [Equation]

Original: where e is a vector describing excitatory neuron activities, i the inhibitory neurons, and ∆ is the learning rate.
Revised: where e is a vector describing excitatory neuron activities, i the inhibitory neurons, and ∆ is the learning rate.

Original: The excitation and inhibition time constants are τ a
Revised: The excitation and inhibition time constants are τ a

Original: and τ i , respectively.
Revised: and τ i , respectively.

Original: L is a smoothing term, b is a firing threshold learnt for each neuron.
Revised: L is a smoothing term, b is a firing threshold learnt for each neuron.

Original: W IE and W EI are the weight matrices from excitatory to inhibition neurons and vice versa; they are fixed and have homogenous weights.
Revised: W IE and W EI are the weight matrices from excitatory to inhibition neurons and vice versa; they are fixed and have homogenous weights.

Original: W inp are the weights that respond to the input x .
Revised: W inp are the weights that respond to the input x .

Original: b i is the firing threshold for the inhibitory interneurons that a fixed scalar and can be thought of as resulting in a noisy approximate k value.
Revised: b i is the firing threshold for the inhibitory interneurons that a fixed scalar and can be thought of as resulting in a noisy approximate k value.

Original: During training, we use a Hebbian learning rule (not given) to train W inp and run the ODEs shown to converge to approximately k neurons remaining on.
Revised: During training, we use a Hebbian learning rule (not given) to train W inp and run the ODEs shown to converge to approximately k neurons remaining on.

Original: These dynamics are simplifications because the weights between excitatory and inhbitory neurons are fixed and homogenous.
Revised: These dynamics are simplifications because the weights between excitatory and inhbitory neurons are fixed and homogenous.

Original: W EI is initialized where every excitatory neuron is connected to each inhibitory with a probability of 90% and strength of 1.
Revised: W EI is initialized where every excitatory neuron is connected to each inhibitory with a probability of 90% and strength of 1.

Original: W IE is initialized with the same connectivity probability but with a strength of 1 / (0 . 9 r I ) where r I is the number of inhibitory neurons.
Revised: W IE is initialized with the same connectivity probability but with a strength of 1 / (0 . 9 r I ) where r I is the number of inhibitory neurons.

Original: This makes the synaptic weights connecting inhibitory to excitatory neurons sum to 1 in expectation.
Revised: This makes the synaptic weights connecting inhibitory to excitatory neurons sum to 1 in expectation.

Original: Putting aside these simplifying assumptions, when we run the model and analyse the number of neurons that remain active, it is approximately a constant k neurons as shown in Fig. 12a.
Revised: Putting aside these simplifying assumptions, when we run the model and analyse the number of neurons that remain active, it is approximately a constant k neurons as shown in Fig. 12a.

Original: Given the inhibitory activation threshold b i , approximately 20% of the neurons remain on for each input giving k ≈ 200 .
Revised: Given the inhibitory activation threshold b i , approximately 20% of the neurons remain on for each input giving k ≈ 200 .

Original: We also show how the average activity of the excitatory neurons evolves over time in Fig.
Revised: We also show how the average activity of the excitatory neurons evolves over time in Fig.

Original: 12b to emphasize that the inhibitory interneuron does not have to make a single guess for how to correctly inhibit all but k neurons.
Revised: 12b to emphasize that the inhibitory interneuron does not have to make a single guess for how to correctly inhibit all but k neurons.

Original: Instead, this is a dynamic process defined by the ODEs of
Revised: Instead, this is a dynamic process defined by the ODEs of

Original: Eq. 13.
Revised: Eq. 13.

Original: We also looked for our SDMLP at the k + 1 -th highest activity value used by the inhibitory neuron to implement Top-K and how this relates to the total unweighted sum of neuron activations entering the inhibitory interneuron.
Revised: We also looked for our SDMLP at the k + 1 -th highest activity value used by the inhibitory neuron to implement Top-K and how this relates to the total unweighted sum of neuron activations entering the inhibitory interneuron.

Original: After the GABA switch when neurons tile the data manifold and form subnetworks, this relationship is largely linear where the k + 1 -th highest activity value is the total sum of all neuron activations before inhibition, divided by the total number of neurons.
Revised: After the GABA switch when neurons tile the data manifold and form subnetworks, this relationship is largely linear where the k + 1 -th highest activity value is the total sum of all neuron activations before inhibition, divided by the total number of neurons.

Original: This is again a simplification by assuming all input and output weights are homogenous and equal to one.
Revised: This is again a simplification by assuming all input and output weights are homogenous and equal to one.

Original: However, again, if the interneuron can modify its pre and post synaptic strengths with the same degree of precision displayed by horizontal cells (Sterling & Laughlin, 2015) then these results support the possibility of the interneuron approximating Top-K, using a simple function to respond to the sum of its inputs.
Revised: However, again, if the interneuron can modify its pre and post synaptic strengths with the same degree of precision displayed by horizontal cells (Sterling & Laughlin, 2015) then these results support the possibility of the interneuron approximating Top-K, using a simple function to respond to the sum of its inputs.

Original: C. O PTIMIZED T OP -K
Revised: C. O PTIMIZED T OP -K

Original: The k value is a parameter which must be chosen in our model and can significantly affect performance.
Revised: The k value is a parameter which must be chosen in our model and can significantly affect performance.

Original: We present empirical results for ablating k on both pretraining and continual learning across a number of different data manifolds.
Revised: We present empirical results for ablating k on both pretraining and continual learning across a number of different data manifolds.

Original: To summarize our findings:
Revised: To summarize our findings:

Original: • A larger k gives better pretraining but worse continual learning and vice versa.
Revised: • A larger k gives better pretraining but worse continual learning and vice versa.

Original: • k depends upon the complexity of the data manifold and number of neurons,
Revised: • k depends upon the complexity of the data manifold and number of neurons,

Original: r . • SDM will approximately tile whatever data manifold it is given.
Revised: r . • SDM will approximately tile whatever data manifold it is given.

Original: Joint training of SDM with other model components such as a CNN will improve pretraining.
Revised: Joint training of SDM with other model components such as a CNN will improve pretraining.

Original: However, learning a non static manifold in this joint training setting is difficult for SDM.
Revised: However, learning a non static manifold in this joint training setting is difficult for SDM.

Original: Optimal k values for Pretraining or Continual Learning?
Revised: Optimal k values for Pretraining or Continual Learning?

Original: Sparsity in the SDM setting presents a fundamental tradeoff between performing well in continual learning (CL) and non continual learning (NCL) settings.
Revised: Sparsity in the SDM setting presents a fundamental tradeoff between performing well in continual learning (CL) and non continual learning (NCL) settings.

Original: This tradeoff exists because sparsity limits the representational capacity of the model and should be expected to reduce NCL accuracy.
Revised: This tradeoff exists because sparsity limits the representational capacity of the model and should be expected to reduce NCL accuracy.

Original: Meanwhile, the more sparse the model is, the better it is able to form subnetworks that are not overwritten by future CL tasks.
Revised: Meanwhile, the more sparse the model is, the better it is able to form subnetworks that are not overwritten by future CL tasks.

Original: Additionally, SDM will tile the data manifold it is given, which depending on the manifold, can be poorly correlated with maximizing classification accuracy.
Revised: Additionally, SDM will tile the data manifold it is given, which depending on the manifold, can be poorly correlated with maximizing classification accuracy.

Original: [Figure]
Revised: [Figure]

Original: We see this tradeoff in Figure 13 that summarizes ablating k during pretraining on ImageNet32 embeddings from the ConvMixer and testing on Split CIFAR10.
Revised: We see this tradeoff in Fig. 13 that summarizes ablating k during pretraining on ImageNet32 embeddings from the ConvMixer and testing on Split CIFAR10.

Original: As the k value decreases, continual learning accuracy improves while pretraining accuracy declines.
Revised: As the k value decreases, continual learning accuracy improves while pretraining accuracy declines.

Original: However, it is unclear why performance on the original ImageNet32 training is parabolic, slightly improving for k < 8 .
Revised: However, it is unclear why performance on the original ImageNet32 training is parabolic, slightly improving for k < 8 .

Original: While we are indifferent to the NCL pretraining task here, using it as a way for the neurons to specialize across an arbitrary manifold of real-world images, it is worth acknowledging the performance is worse.
Revised: While we are indifferent to the NCL pretraining task here, using it as a way for the neurons to specialize across an arbitrary manifold of real-world images, it is worth acknowledging the performance is worse.

Original: That this performance decline is caused by sparsity is even more evident during a single training run as GABA switches and k is slowly annealed.
Revised: That this performance decline is caused by sparsity is even more evident during a single training run as GABA switches and k is slowly annealed.

Original: Fig.
Revised: Fig.

Original: 14a shows this for smaller k values where performance declines.
Revised: 14a shows this for smaller k values where performance declines.

Original: To emphasize how the optimal k value changes as a function of the number of neurons and data manifold, we describe a number of additional results:
Revised: To emphasize how the optimal k value changes as a function of the number of neurons and data manifold, we describe a number of additional results:

Original: For the same ConvMixer embeddings of ImageNet32 pretraining and Split CIFAR10 testing, if we use 10K neurons instead of 1K then there is the same linear decline in pretraining performance as in Fig.
Revised: For the same ConvMixer embeddings of ImageNet32 pretraining and Split CIFAR10 testing, if we use 10K neurons instead of 1K then there is the same linear decline in pretraining performance as in Fig.

Original: 13 but k = 5 outperforms all k ≤ 50 , it is also the best for continual learning.
Revised: 13 but k = 5 outperforms all k ≤ 50 , it is also the best for continual learning.

Original: Meanwhile, k = 1 becomes the worst for pretraining and the second best for continual learning.
Revised: Meanwhile, k = 1 becomes the worst for pretraining and the second best for continual learning.

Original: With the same setup again but operating directly on pixels without using the ConvMixer embeddings, k = 10 is the best for pretraining but k = 1 is the best at continual learning as shown in Fig.
Revised: With the same setup again but operating directly on pixels without using the ConvMixer embeddings, k = 10 is the best for pretraining but k = 1 is the best at continual learning as shown in Fig.

Original: 18 of
Revised: 18 of

Original: Appendix E.1.
Revised: App. E.1.

Original: In all of the above cases we are given a static manifold for SDM to learn.
Revised: In all of the above cases we are given a static manifold for SDM to learn.

Original: This always results in worse pretraining accuracy than an equivalent ReLU MLP, even when using large k values.
Revised: This always results in worse pretraining accuracy than an equivalent ReLU MLP, even when using large k values.

Original: This is because the ReLU MLP is not constrained to the data manifold, having a bias term and no L 2 normalization.
Revised: This is because the ReLU MLP is not constrained to the data manifold, having a bias term and no L 2 normalization.

Original: This allows it to learn weights and biases that better maximize NCL validation accuracy.
Revised: This allows it to learn weights and biases that better maximize NCL validation accuracy.

Original: [Figure]
Revised: [Figure]

Original: If we perform joint training of the ConvMixer and SDM module then the ConvMixer can learn to create a manifold for SDM to tile that does maximize validation accuracy, performing on par with the ReLU MLP.
Revised: If we perform joint training of the ConvMixer and SDM module then the ConvMixer can learn to create a manifold for SDM to tile that does maximize validation accuracy, performing on par with the ReLU MLP.

Original: This is what happens in a test where we train on the whole CIFAR10 dataset in the NCL setting, as long as k ≥ 250 , but independent of if there are 1,000 or 10,000 neurons in the SDM layer.
Revised: This is what happens in a test where we train on the whole CIFAR10 dataset in the NCL setting, as long as k ≥ 250 , but independent of if there are 1,000 or 10,000 neurons in the SDM layer.

Original: This result is shown in Fig. 14a and suggests that the ReLU network only needs at least 250 neurons in its final layer to backpropagate gradients and achieve high prediction accuracy.
Revised: This result is shown in Fig. 14a and suggests that the ReLU network only needs at least 250 neurons in its final layer to backpropagate gradients and achieve high prediction accuracy.

Original: There is evidence that artificial neural networks are overparameterized at the start of training and functionally sparse, as supported by results from network pruning, most notably the Lottery Ticket
Revised: There is evidence that artificial neural networks are overparameterized at the start of training and functionally sparse, as supported by results from network pruning, most notably the Lottery Ticket

Original: Hypothesis (Frankle & Carbin, 2019).
Revised: Hypothesis (Frankle & Carbin, 2019).

Original: However, this result is again manifold dependent whereby training instead on ImageNet32, even with k = 2 , 500 and 10,000 neurons still harms performance compared to a ReLU network.
Revised: However, this result is again manifold dependent whereby training instead on ImageNet32, even with k = 2 , 500 and 10,000 neurons still harms performance compared to a ReLU network.

Original: We believe this is because ImageNet32 has a dramatically more complex data manifold with ∼ 1.2M images from 1,000 different classes, compared to 50,000 images in 10 classes for CIFAR10.
Revised: We believe this is because ImageNet32 has a dramatically more complex data manifold with ∼ 1.2M images from 1,000 different classes, compared to 50,000 images in 10 classes for CIFAR10.

Original: This means that even a k of 2500 is too small and harms the model’s representational capacity.
Revised: This means that even a k of 2500 is too small and harms the model’s representational capacity.

Original: An additional difficulty with joint training is the timing of the GABA switch to avoid dead neurons.
Revised: An additional difficulty with joint training is the timing of the GABA switch to avoid dead neurons.

Original: This is because the manifold SDM learns is dynamically changing over time and we believe it explains the neuron death that is inversely proportional to k shown in Fig. 14b.
Revised: This is because the manifold SDM learns is dynamically changing over time and we believe it explains the neuron death that is inversely proportional to k shown in Fig. 14b.

Original: We leave it to future work to further investigate optimal k values and joint training versus using frozen, pretrained models.
Revised: We leave it to future work to further investigate optimal k values and joint training versus using frozen, pretrained models.

Original: We are optimistic that the best way to resolve the tradeoff from sparsity limiting representational capacity is to make the network layers wider rather than the number of neurons that can be active, k , fewer.
Revised: We are optimistic that the best way to resolve the tradeoff from sparsity limiting representational capacity is to make the network layers wider rather than the number of neurons that can be active, k , fewer.

Original: This is feasible because sparse activations are computationally cheap – while computing the k most active neurons requires multiplying the input with each neuron (this can be parallelized) and then sorting them, a constant k << r neurons produce outputs and receive gradient updates.
Revised: This is feasible because sparse activations are computationally cheap – while computing the k most active neurons requires multiplying the input with each neuron (this can be parallelized) and then sorting them, a constant k << r neurons produce outputs and receive gradient updates.

Original: Computing which neurons are active would also be cheaper with sparse weights (Jaeckel, 1989a; Ahmad & Scheinkman, 2019).
Revised: Computing which neurons are active would also be cheaper with sparse weights (Jaeckel, 1989a; Ahmad & Scheinkman, 2019).

Original: SDM Theory to derive optimal k values
Revised: SDM Theory to derive optimal k values

Original: Before concluding this section, we want to flag to readers that SDM, with a few simplifying assumptions, is able to analytically derive optimal k values under a few different conditions of optimality: (i) Optimising the signal to noise ratio of every pattern; (ii) Optimizing for the maximum number of patterns that can be stored, within a certain retrieval probability; (iii) Maximizing the distance a query can be from its target pattern while still converging correctly.
Revised: Before concluding this section, we want to flag to readers that SDM, with a few simplifying assumptions, is able to analytically derive optimal k values under a few different conditions of optimality: (i) Optimising the signal to noise ratio of every pattern; (ii) Optimizing for the maximum number of patterns that can be stored, within a certain retrieval probability; (iii) Maximizing the distance a query can be from its target pattern while still converging correctly.

Original: These derivations and further discussion be found in (Kanerva, 1993; Bricken & Pehlevan, 2021).
Revised: These derivations and further discussion be found in (Kanerva, 1993; Bricken & Pehlevan, 2021).

Original: While flagging that these analytical results exist and are interesting, there are all concerned with maximizing the information content of the patterns stored that is most relevant to a reconstruction task.
Revised: While flagging that these analytical results exist and are interesting, there are all concerned with maximizing the information content of the patterns stored that is most relevant to a reconstruction task.

Original: In this paper, we care about two different objective functions somewhat uncorrelated with maximizing information content: (i) Classification accuracy, where better performance can be achieved without trying to model the data manifold; (ii) Continual learning, where while we care about modelling the data manifold, we also want the formation of unique subnetworks that can output the correct classification label.
Revised: In this paper, we care about two different objective functions somewhat uncorrelated with maximizing information content: (i) Classification accuracy, where better performance can be achieved without trying to model the data manifold; (ii) Continual learning, where while we care about modelling the data manifold, we also want the formation of unique subnetworks that can output the correct classification label.

Original: In addition, SDM’s analytical results assume that the patterns are random.
Revised: In addition, SDM’s analytical results assume that the patterns are random.

Original: For correlated real-world datasets, we need ways to quantify the complexity of the data manifold and a full exploration of this problem and solutions is beyond the scope of this work.
Revised: For correlated real-world datasets, we need ways to quantify the complexity of the data manifold and a full exploration of this problem and solutions is beyond the scope of this work.

Original: D S TALE M OMENTUM
Revised: D S TALE M OMENTUM

Original: Imagine a neuron is activated for three training batches, then inactive for the next 10 batches, and finally activated two more times.
Revised: Imagine a neuron is activated for three training batches, then inactive for the next 10 batches, and finally activated two more times.

Original: During the first set of activations there are no problems, the neuron will update its weights and its momentum term, using up to date gradient information.
Revised: During the first set of activations there are no problems, the neuron will update its weights and its momentum term, using up to date gradient information.

Original: During inactivity, problems begin as the inactive neuron will still update its weights and do so using an increasingly out of date momentum term.
Revised: During inactivity, problems begin as the inactive neuron will still update its weights and do so using an increasingly out of date momentum term.

Original: Problems continue when this neuron is activated again because the momentum term decays slowly and will significantly boost the gradient in an out of date direction.
Revised: Problems continue when this neuron is activated again because the momentum term decays slowly and will significantly boost the gradient in an out of date direction.

Original: These stale momenta are especially harmful in the Top-K competitive activation setting where a neuron needs to be the k most active for at least one input to continue receiving gradient updates, otherwise it will permanently die.
Revised: These stale momenta are especially harmful in the Top-K competitive activation setting where a neuron needs to be the k most active for at least one input to continue receiving gradient updates, otherwise it will permanently die.

Original: Using Stochastic Gradient Descent (SGD) that is free from any momentum term removes this problem.
Revised: Using Stochastic Gradient Descent (SGD) that is free from any momentum term removes this problem.

Original: SGD with momentum works worse than SGD without momentum, but still better than Adam or RMSProp (Kingma & Ba, 2015; Geoffrey Hinton, 2012;
Revised: SGD with momentum works worse than SGD without momentum, but still better than Adam or RMSProp (Kingma & Ba, 2015; Geoffrey Hinton, 2012;

Original: Goodfellow et al., 2015).
Revised: Goodfellow et al., 2015).

Original: We discovered the Stale Momentum effect when training our models directly on CIFAR10 images that have a much more complex data manifold than the ConvMixer embeddings.
Revised: We discovered the Stale Momentum effect when training our models directly on CIFAR10 images that have a much more complex data manifold than the ConvMixer embeddings.

Original: Figure 15 shows that after the GABA switch, when Top-K is fully implemented around epoch 200, Adam and RMSProp continue killing off neurons (Kingma & Ba, 2015; Geoffrey Hinton, 2012).
Revised: Fig. 15 shows that after the GABA switch, when Top-K is fully implemented around epoch 200, Adam and RMSProp continue killing off neurons (Kingma & Ba, 2015; Geoffrey Hinton, 2012).

Original: These neurons are dead in that they are never activated by any training or validation data.
Revised: These neurons are dead in that they are never activated by any training or validation data.

Original: Our results are robust across learning rates for the different optimizers.
Revised: Our results are robust across learning rates for the different optimizers.

Original: While these dead neurons from Adam and RMSProp don’t appear when pretraining on ImageNet embeddings, the Stale Momentum problem still leads to incorrect gradient updates that harm continual learning as shown in the ablations of Table 2.
Revised: While these dead neurons from Adam and RMSProp don’t appear when pretraining on ImageNet embeddings, the Stale Momentum problem still leads to incorrect gradient updates that harm continual learning as shown in the ablations of Table 2.

Original: This is because fewer neurons are covering the data manifold and able to retain memories of separate tasks.
Revised: This is because fewer neurons are covering the data manifold and able to retain memories of separate tasks.

Original: Notably, a ReLU network trained with Adam or RMSProp also kills off up to 95% of its neurons with only minor effects on train and validation accuracy when trained directly on CIFAR10 pixels.
Revised: Notably, a ReLU network trained with Adam or RMSProp also kills off up to 95% of its neurons with only minor effects on train and validation accuracy when trained directly on CIFAR10 pixels.

Original: SGDM and SGD again do not kill off any neurons for this network.
Revised: SGDM and SGD again do not kill off any neurons for this network.

Original: [Figure]
Revised: [Figure]

Original: We logged the gradients of neuron weights during training to confirm that neurons are receiving gradient updates even while they are dead as shown in Figure 16a.
Revised: We logged the gradients of neuron weights during training to confirm that neurons are receiving gradient updates even while they are dead as shown in Fig. 16a.

Original: This figure is initially confusing because, while Adam and RMSProp kill off the most neurons in Fig.
Revised: This figure is initially confusing because, while Adam and RMSProp kill off the most neurons in Fig.

Original: 15, it is Adam and SGDM that keep updating dead neurons in Fig.
Revised: 15, it is Adam and SGDM that keep updating dead neurons in Fig.

Original: 16a (Kingma & Ba, 2015; Geoffrey Hinton, 2012).
Revised: 16a (Kingma & Ba, 2015; Geoffrey Hinton, 2012).

Original: Looking at every gradient update applied to a single neuron right after GABA switches in Fig.
Revised: Looking at every gradient update applied to a single neuron right after GABA switches in Fig.

Original: 16b provides a different perspective.
Revised: 16b provides a different perspective.

Original: Here, the highest variance and largest gradient magnitudes (as quantified by their L 2 norm) are produced by RMSProp and Adam.
Revised: Here, the highest variance and largest gradient magnitudes (as quantified by their L 2 norm) are produced by RMSProp and Adam.

Original: These gradient spikes appear particularly in cases where the neuron is dead for some time and then activated again.
Revised: These gradient spikes appear particularly in cases where the neuron is dead for some time and then activated again.

Original: To investigate further, we implemented each optimizer in a toy environment tracking a single weight that receives sparse gradient updates to see how the moving averages respond.
Revised: To investigate further, we implemented each optimizer in a toy environment tracking a single weight that receives sparse gradient updates to see how the moving averages respond.

Original: 14 We were able to reproduce the much larger gradient spikes displayed by Adam and RMSProp upon the first few gradient updates after a period of quiescence.
Revised: 15 We were able to reproduce the much larger gradient spikes displayed by Adam and RMSProp upon the first few gradient updates after a period of quiescence.

Original: These results are shown in Fig.
Revised: These results are shown in Fig.

Original: 17a, where we show the ∆ Optimizer term for each optimizer, independent of the learning rate.
Revised: 17a, where we show the ∆ Optimizer term for each optimizer, independent of the learning rate.

Original: Initializing their respective terms shown in Eqs.
Revised: Initializing their respective terms shown in Eqs.

Original: 14 with zeros, we first introduce two gradient updates and then have periods of quiescence before injecting gradient updates.
Revised: 14 with zeros, we first introduce two gradient updates and then have periods of quiescence before injecting gradient updates.

Original: In Fig.
Revised: In Fig.

Original: 17a we show four different gradient injection periods and that RMSProp has the largest response, followed by Adam and then SGDM, reproducing the gradient spikes seen in Fig. 16b.
Revised: 17a we show four different gradient injection periods and that RMSProp has the largest response, followed by Adam and then SGDM, reproducing the gradient spikes seen in Fig. 16b.

Revised: Fig.

Original: Figure 17b explains these larger gradients by plotting the Adam and RMSProp numerator and denominators for only two gradient injection periods.
Revised: 17b explains these larger gradients by plotting the Adam and RMSProp numerator and denominators for only two gradient injection periods.

Original: Note in particular that for Adam, its numerator m t (dark orange) declines somewhat quickly, having a decay of β = 0 .
Revised: Note in particular that for Adam, its numerator m t (dark orange) declines somewhat quickly, having a decay of β = 0 .

Original: 9 , however, its denomina- tor √ v t (light orange) has a much slower decay of β 2 = 0 . 999 , staying large for much longer and causing the new gradient inputs to explode in size.
Revised: 9 , however, its denominator √ v t (light orange) has a much slower decay of β 2 = 0 . 999 , staying large for much longer and causing the new gradient inputs to explode in size.

Original: Finally, Figure 17c quantifies how large the gaps in gradient update magnitude are between the three optimizers and the actual gradient when we vary the time between gradient injections.
Revised: Finally, Fig. 17c quantifies how large the gaps in gradient update magnitude are between the three optimizers and the actual gradient when we vary the time between gradient injections.

Original: To summarize, we believe it is more the gradient explosions after quiescence rather than updating of neurons currently dead that causes Adam and RMSProp to kill off neurons as in Fig.
Revised: To summarize, we believe it is more the gradient explosions after quiescence rather than updating of neurons currently dead that causes Adam and RMSProp to kill off neurons as in Fig.

Original: The denominators present in these optimizers, given in Eqs.
Revised: The denominators present in these optimizers, given in Eqs.

Original: 16 and 17 are to blame because they are such slow moving averages and inflate the sparse gradients.
Revised: 16 and 17 are to blame because they are such slow moving averages and inflate the sparse gradients.

Original: The update rules for SGDM, Adam, and RMSProp are as follows where we let λ be the learning rate, θ t are the parameters at time t , g t is the gradient, and γ = 0 .
Revised: The update rules for SGDM, Adam, and RMSProp are as follows where we let λ be the learning rate, θ t are the parameters at time t , g t is the gradient, and γ = 0 .

Original: 9 β 1 = 0 .
Revised: 9 β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , α = 0 .are
Revised: 999 , α = 0 .are

Original: hyperparameters for the various algorithms with their standard values.
Revised: hyperparameters for the various algorithms with their standard values.

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that these results are robust for any gradient updates g t < 1 , otherwise SGDM has larger gradient spikes than Adam.
Revised: Note that these results are robust for any gradient updates g t < 1 , otherwise SGDM has larger gradient spikes than Adam.

Original: However, this is never the case in our training regime because of the Lnormalization term for our weights and inputs which means that no weight or neuron activation value is ever larger than 1.
Revised: However, this is never the case in our training regime because of the Lnormalization term for our weights and inputs which means that no weight or neuron activation value is ever larger than 1.

Original: If this were not the case then we predict that SGDM would have sufficiently large jumps in its gradients to also result in dead neurons.
Revised: If this were not the case then we predict that SGDM would have sufficiently large jumps in its gradients to also result in dead neurons.

Original: Practical Takeaways Below we give our reasoning for why SGD is the best solution but needs careful hyperparameter tuning of the learning rate and GABA switch s .
Revised: Practical Takeaways Below we give our reasoning for why SGD is the best solution but needs careful hyperparameter tuning of the learning rate and GABA switch s .

Original: Sparse optimizers (Spa) are another alternative that should be investigated in future work.
Revised: Sparse optimizers (Spa) are another alternative that should be investigated in future work.

Original: Because SGD has no momentum term it does not suffer from stale momenta and is the most principled solution.
Revised: Because SGD has no momentum term it does not suffer from stale momenta and is the most principled solution.

Original: However, lacking a momentum term, SGD is also the most sensitive to learning rate choice and can introduce either slow or poor convergence if it is too high or low, respectively.
Revised: However, lacking a momentum term, SGD is also the most sensitive to learning rate choice and can introduce either slow or poor convergence if it is too high or low, respectively.

Original: Setting the learning rate too low is particularly problematic in our setting with the excitatory period of the GABA switch because neurons can fail to receive enough gradient updates to move onto the data manifold, resulting in dead neurons.
Revised: Setting the learning rate too low is particularly problematic in our setting with the excitatory period of the GABA switch because neurons can fail to receive enough gradient updates to move onto the data manifold, resulting in dead neurons.

Original: In Fig.
Revised: In Fig.

Original: 11 of Appendix B.3 we show how a learning rate that is too low will result in dead neurons that never learn the manifold before the GABA switch occurs.
Revised: 11 of App. B.3 we show how a learning rate that is too low will result in dead neurons that never learn the manifold before the GABA switch occurs.

Original: Given these hyperparameter tuning requirements for SGD, we recommend using SGDM to pretrain
Revised: Given these hyperparameter tuning requirements for SGD, we recommend using SGDM to pretrain

Original: SDMLP while using SGD for continual learning.
Revised: SDMLP while using SGD for continual learning.

Original: SGDM pretraining does not result in dead neurons as shown in Fig.
Revised: SGDM pretraining does not result in dead neurons as shown in Fig.

Original: 15 and is more robust to choice of learning rate.
Revised: 15 and is more robust to choice of learning rate.

Original: For example in this instance of SGDM on CIFAR10 pixels, varying the learning rate from 0.01 to 0.09 did not affect either
Revised: For example in this instance of SGDM on CIFAR10 pixels, varying the learning rate from 0.01 to 0.09 did not affect either

Original: [Figure]
Revised: [Figure]

Original: Note that SGDM without a denominator term is the only optimizer that over longer periods of time stops amplifying the gradient value.
Revised: Note that SGDM without a denominator term is the only optimizer that over longer periods of time stops amplifying the gradient value.

Original:  generalize for  that are convergence speed or kill off any additional neurons.
Revised: All generalize for gradient that are convergence speed or kill off any additional neurons.

Original: Meanwhile, as shown in the ablations of Table2, in the continual learning setting, SGD is the best choice.
Revised: Meanwhile, as shown in the ablations of Table in the continual learning setting, SGD is the best choice.

Original: Another alternative are sparse optimizers like “sparse Adam” which does not apply gradient updates to any neuron that is dead (Spa).
Revised: Another alternative are sparse optimizers like “sparse Adam” which does not apply gradient updates to any neuron that is dead (Spa).

Original: However, failing to update the momentum term in a novel way will theoretically still result in what we believe to be the larger problem of exploding re-activated neuron gradients.
Revised: However, failing to update the momentum term in a novel way will theoretically still result in what we believe to be the larger problem of exploding re-activated neuron gradients.

Original: The only way to use sparse Adam in Pytorch is to implement sparse weight layers and we leave it to future work to empirically test sparse momentum based optimizers.
Revised: The only way to use sparse Adam in Pytorch is to implement sparse weight layers and we leave it to future work to empirically test sparse momentum based optimizers.

Original: This section lacks many citations because we are unaware of existing literature around other models in the sparse activation regime, using either Top-K or Mixture of Experts (Fedus et al., 2021;
Revised: This section lacks many citations because we are unaware of existing literature around other models in the sparse activation regime, using either Top-K or Mixture of Experts (Fedus et al., 2021;