Original: Multilayer Neuromodulated Architectures for Memory-Constrained Online Continual Learning
Revised: Multilayer Neuromodulated Architectures for Memory-Constrained Online Continual Learning

Original: Abstract
Revised: Abstract

Original: We focus on the problem of how to achieve online continual learning under memory-constrained conditions where the input data may not be known a priori .
Revised: We focus on the problem of how to achieve online continual learning under memory-constrained conditions where the input data may not be known a priori .

Original: These constraints are relevant in edge computing scenarios.
Revised: These constraints are relevant in edge computing scenarios.

Original: We have developed an architecture where input processing over data streams and online learning are integrated in a single recurrent network architecture.
Revised: We have developed an architecture where input processing over data streams and online learning are integrated in a single recurrent network architecture.

Original: This allows us to cast metalearning optimization as a mixed-integer optimization problem, where different synaptic plasticity algorithms and feature extraction layers can be swapped out and their hyperparameters are optimized to identify optimal architectures for different sets of tasks.
Revised: This allows us to cast metalearning optimization as a mixed-integer optimization problem, where different synaptic plasticity algorithms and feature extraction layers can be swapped out and their hyperparameters are optimized to identify optimal architectures for different sets of tasks.

Original: We utilize a Bayesian optimization method to search over a design space that spans multiple learning algorithms, their speciﬁc hyperparameters, and feature extraction layers.
Revised: We utilize a Bayesian optimization method to search over a design space that spans multiple learning algorithms, their speciﬁc hyperparameters, and feature extraction layers.

Original: We demonstrate our approach for online nonincremental and class-incremental learning tasks.
Revised: We demonstrate our approach for online nonincremental and class-incremental learning tasks.

Original: Our optimization algorithm ﬁnds conﬁgurations that achieve superior continual learning performance on Split-MNIST and Permuted-MNIST data as compared with other memory-constrained learning approaches, and it matches that of the state-of-the-art memory replay-based approaches without explicit data storage and replay.
Revised: Our optimization algorithm ﬁnds conﬁgurations that achieve superior continual learning performance on Split-MNIST and Permuted-MNIST data as compared with other memory-constrained learning approaches, and it matches that of the state-of-the-art memory replay-based approaches without explicit data storage and replay.

Original: Our approach allows us to explore the transferability of optimal learning conditions to tasks and datasets that have not been previously seen.
Revised: Our approach allows us to explore the transferability of optimal learning conditions to tasks and datasets that have not been previously seen.

Original: We demonstrate that the accuracy of our transfer metalearning across datasets can be largely explained through a transfer coefﬁcient that can be based on metrics of dimensionality and distance between datasets.
Revised: We demonstrate that the accuracy of our transfer metalearning across datasets can be largely explained through a transfer coefﬁcient that can be based on metrics of dimensionality and distance between datasets.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn and generalize from a few examples is present on a wide range of species, from the fruit ﬂy to humans.
Revised: The ability to learn and generalize from a few examples is present on a wide range of species, from the fruit ﬂy to humans.

Original: It is therefore not surprising that biological systems have served as inspiration for the exploration of continual or incremental learning.
Revised: It is therefore not surprising that biological systems have served as inspiration for the exploration of continual or incremental learning.

Original: How to achieve the right balance between plasticity and consolidation, and therefore how to avoid catastrophic forgetting, has long been one of the central areas of research in the area of continual learning.
Revised: How to achieve the right balance between plasticity and consolidation, and therefore how to avoid catastrophic forgetting, has long been one of the central areas of research in the area of continual learning.

Original: However, the design of systems intended for edge computing brings up an additional set of constraints, such as how to design systems that have to learn inputs that are not known during the design phase, how to achieve high accuracy when the system has no external way of replaying data (i.e., stream processing), or what the tradeoff is between performance and size.
Revised: However, the design of systems intended for edge computing brings up an additional set of constraints, such as how to design systems that have to learn inputs that are not known during the design phase, how to achieve high accuracy when the system has no external way of replaying data (i.e., stream processing), or what the tradeoff is between performance and size.

Original: These resource constraints may cause performance penalties with respect to online and continual learning strategies that are not bounded by memory and that keep a detailed record of past performance and data.
Revised: These resource constraints may cause performance penalties with respect to online and continual learning strategies that are not bounded by memory and that keep a detailed record of past performance and data.

Original: On the other hand, these constraints are also present in biological systems, and yet they excel in many continual learning tasks.
Revised: On the other hand, these constraints are also present in biological systems, and yet they excel in many continual learning tasks.

Original: Neuroinspired approaches can therefore provide useful insights that complement other machine learning approaches.
Revised: Neuroinspired approaches can therefore provide useful insights that complement other machine learning approaches.

Original: One of the salient features of the central nervous systems is its heterogeneity in terms of architecture, types of neurons, and neurochemistry, including where, when, and how learning takes place within the brain.
Revised: One of the salient features of the central nervous systems is its heterogeneity in terms of architecture, types of neurons, and neurochemistry, including where, when, and how learning takes place within the brain.

Original: If we focus on synaptic plasticity mechanisms, we see a wide diversity, not only in the hyperparameters of a speciﬁc mechanism or in where learning takes place, but also in the nature of the synaptic plasticity mechanism itself.
Revised: If we focus on synaptic plasticity mechanisms, we see a wide diversity, not only in the hyperparameters of a speciﬁc mechanism or in where learning takes place, but also in the nature of the synaptic plasticity mechanism itself.

Original: For instance, while Hebbian rules and spike timing-dependent plasticity have attracted the bulk of the attention in the machine learning and neuromorphic computing communities, anti-Hebbian and nonHebbian rules abound (Dayan, 2001).
Revised: For instance, while Hebbian rules and spike timing-dependent plasticity have attracted the bulk of the attention in the machine learning and neuromorphic computing communities, anti-Hebbian and nonHebbian rules abound (Dayan, 2001).

Original: For instance, in some key learning centers researchers have showed that synaptic plasticity is independent of postsynaptic activity (Hige et al., 2015).
Revised: For instance, in some key learning centers researchers have showed that synaptic plasticity is independent of postsynaptic activity (Hige et al., 2015).

Original: Moreover, variations in the local learning rules have been found even in areas that are otherwise morphologically equivalent but that are assigned to different tasks.
Revised: Moreover, variations in the local learning rules have been found even in areas that are otherwise morphologically equivalent but that are assigned to different tasks.

Original: Clearly, if we want to develop robust continual learning approaches, we need to pay attention to this diversity and develop a better understanding of the interplay between heterogeneity and continual learning across very different tasks, which is often overlooked in continual learning across diverse tasks.
Revised: Clearly, if we want to develop robust continual learning approaches, we need to pay attention to this diversity and develop a better understanding of the interplay between heterogeneity and continual learning across very different tasks, which is often overlooked in continual learning across diverse tasks.

Original: To this end, we make the following contributions
Revised: To this end, we make the following contributions

Original: • We propose a neuromorphic architecture that consists of multiple layers for feature extraction and neuromodulated supervised learning.
Revised: • We propose a neuromorphic architecture that consists of multiple layers for feature extraction and neuromodulated supervised learning.

Original: It incorporates diverse synaptic plasticity mechanisms, and hence is capable of online continual learning without explicit data storage and replay.
Revised: It incorporates diverse synaptic plasticity mechanisms, and hence is capable of online continual learning without explicit data storage and replay.

Original: • We cast the problem of metalearning task-speciﬁc architecture choice as a mixed-integer black-box optimization problem and develop a technique based on transfer coefﬁcients to explain the accuracy differences when the metalearned conﬁgurations are transferred across datasets and tasks.
Revised: • We cast the problem of metalearning task-speciﬁc architecture choice as a mixed-integer black-box optimization problem and develop a technique based on transfer coefﬁcients to explain the accuracy differences when the metalearned conﬁgurations are transferred across datasets and tasks.

Original: • We evaluate our approach in an online non-incremental learning and class-incremental learning scenarios and demonstrate that the accuracy values are on par with the shallow networks in the former scenario and outperform other memory-free continual learning approaches while obtaining accuracies similar to those of the memoryreplay-based approaches without the memory.
Revised: • We evaluate our approach in an online non-incremental learning and class-incremental learning scenarios and demonstrate that the accuracy values are on par with the shallow networks in the former scenario and outperform other memory-free continual learning approaches while obtaining accuracies similar to those of the memoryreplay-based approaches without the memory.

Original:  Multilayer Neuromodulated Architecture
Revised:  Multilayer Neuromodulated Architecture

Original: We consider multilayer, recurrent architectures integrating both processing and learning into the networks as shown in
Revised: We consider multilayer, recurrent architectures integrating both processing and learning into the networks as shown in

Original: Fig.
Revised: Fig.

Original: MULTILAYER ARCHITECTURE
Revised: MULTILAYER ARCHITECTURE

Original: [Figure]
Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation][Figure] [Equation] [Figure]
Revised: [Equation]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: 2.1 Processing Component
Revised: 2.1 Processing Component

Original: The processing component comprises a feed-forward neural network.
Revised: The processing component comprises a feed-forward neural network.

Original: It has two layers:
Revised: It has two layers:

Original: Feature extraction layers: It is based on sparse projection ( SPARSE ) of the N -dimensional input into a much larger M -dimensional space.
Revised: Feature extraction layers: It is based on sparse projection ( SPARSE ) of the N -dimensional input into a much larger M -dimensional space.

Original: Each neuron receives K inputs, with K (cid:28) N .
Revised: Each neuron receives K inputs, with K (cid:28) N .

Original: In addition, dynamic thresholding followed by rectiﬁcation ensures that only the most salient features are included in the representation:
Revised: In addition, dynamic thresholding followed by rectiﬁcation ensures that only the most salient features are included in the representation:

Original: [Equation]
Revised: [Equation]

Original: where W e is the sparse projection matrix, µ and σ are the mean and standard deviation of the matrix multiplication product W e u , and β 1 r is a constant controlling the cutoff.
Revised: where W e is the sparse projection matrix, µ and σ are the mean and standard deviation of the matrix multiplication product W e u , and β 1 r is a constant controlling the cutoff.

Original: This approach highlights correlations between different input channels and represents the broadest possible prior for feature extraction, since it does not assume any spatial dependence or correlations between inputs.
Revised: This approach highlights correlations between different input channels and represents the broadest possible prior for feature extraction, since it does not assume any spatial dependence or correlations between inputs.

Original: This layer is similar to that used in previous models inspired in the cerebellum and the mushroom body, two of the biological instances of this architecture (Litwin-Kumar et al., 2017; Yanguas-Gil, 2019).
Revised: This layer is similar to that used in previous models inspired in the cerebellum and the mushroom body, two of the biological instances of this architecture (Litwin-Kumar et al., 2017; Yanguas-Gil, 2019).

Original: A key distinction is that W e , K , and β 1 r can all be adjusted through our optimization algorithm.
Revised: A key distinction is that W e , K , and β 1 r can all be adjusted through our optimization algorithm.

Original: Neuromodulated learning layer: This comprises one or more layers where supervised learning occurs.
Revised: Neuromodulated learning layer: This comprises one or more layers where supervised learning occurs.

Original: Synaptic weights are treated as ﬁrst-class network elements that can be updated in real time by using a series of local learning rules codiﬁed in the learning component.
Revised: Synaptic weights are treated as ﬁrst-class network elements that can be updated in real time by using a series of local learning rules codiﬁed in the learning component.

Original: 2.2 Learning Component
Revised: 2.2 Learning Component

Original: It comprises the recurrent version of the network, implementing local learning rules.
Revised: It comprises the recurrent version of the network, implementing local learning rules.

Original: These local learning rules are deﬁned as functions:
Revised: These local learning rules are deﬁned as functions:

Original: [Equation]
Revised: [Equation]

Original: where W represents the synaptic weights; x e and x o are the presynaptic and postsynaptic activities, respectively; x m represents a vector of modulatory signals controlling learning in real time; and β l represents a discrete number of hyperparameters that are speciﬁc to a given learning rule.
Revised: where W represents the synaptic weights; x e and x o are the presynaptic and postsynaptic activities, respectively; x m represents a vector of modulatory signals controlling learning in real time; and β l represents a discrete number of hyperparameters that are speciﬁc to a given learning rule.

Original: By using a consistent interface, we can swap out learning rules without having to further modify our architecture.
Revised: By using a consistent interface, we can swap out learning rules without having to further modify our architecture.

Original: This allows us to treat metalearning as a combined architecture and hyperparameter optimization process.
Revised: This allows us to treat metalearning as a combined architecture and hyperparameter optimization process.

Original: The architecture receives real-time feedback on its perfor-
Revised: The architecture receives real-time feedback on its perfor-

Original: mance while learning (i.e., the label).
Revised: mance while learning (i.e., the label).

Original: The modulatory component transforms this feedback into the modulatory signals that are fed into the local learning rules.
Revised: The modulatory component transforms this feedback into the modulatory signals that are fed into the local learning rules.

Original: Here, we use the following synaptic plasticity rules.
Revised: Here, we use the following synaptic plasticity rules.

Original: • Generalized Hebbian model (GEN): the generalized Hebbian model is a modulated version of the covariance rule commonly used in neuroscience.
Revised: • Generalized Hebbian model (GEN): the generalized Hebbian model is a modulated version of the covariance rule commonly used in neuroscience.

Original: The synaptic weight
Revised: The synaptic weight

Original: evolution is given by
Revised: evolution is given by

Original: [Equation]
Revised: [Equation]

Original: It has long been known that this rule is unstable and that clamping of a regularization mechanism needs to be included in order to keep the synaptic weights bounded.
Revised: It has long been known that this rule is unstable and that clamping of a regularization mechanism needs to be included in order to keep the synaptic weights bounded.

Original: This rule is at the core of some of the neuromodulationbased approaches recently developed (Miconi et al., 2018).
Revised: This rule is at the core of some of the neuromodulationbased approaches recently developed (Miconi et al., 2018).

Original: • Oja rule (OJA): a modiﬁcation of the basic Hebb’s rule providing a normalization mechanisms through a ﬁrstorder loss term (Oja, 1982):
Revised: • Oja rule (OJA): a modiﬁcation of the basic Hebb’s rule providing a normalization mechanisms through a ﬁrstorder loss term (Oja, 1982):

Original: [Equation]
Revised: [Equation]

Original: • MSE, non-Hebbian rule: It is based on recent experimental results on synaptic plasticity mechanisms in the mushroom body (Hige et al., 2015).
Revised: • MSE, non-Hebbian rule: It is based on recent experimental results on synaptic plasticity mechanisms in the mushroom body (Hige et al., 2015).

Original: The key assumptions are that learning is independent of postsynaptic activity and that instead postsynaptic activity modulates regulates synaptic plasticity:
Revised: The key assumptions are that learning is independent of postsynaptic activity and that instead postsynaptic activity modulates regulates synaptic plasticity:

Original: [Equation]
Revised: [Equation]

Original: This rule is consistent with the MSE cost function used in stochastic gradient descent methods (Yanguas-Gil et al., 2019).
Revised: This rule is consistent with the MSE cost function used in stochastic gradient descent methods (Yanguas-Gil et al., 2019).

Original: These are just
Revised: These are just

Original: a few of the many nonequivalent possible
Revised: a few of the many nonequivalent possible

Original: formulations of
Revised: formulations of

Original: synaptic plasticity
Revised: synaptic plasticity

Original: rules
Revised: rules

Original: Madireddy
Revised: Madireddy

Original: et al., 2019)
Revised: et al., 2019)

Original: We have treated these models
Revised: We have treated these models

Original: as categorical variables that
Revised: as categorical variables that

Original: can
Revised: can

Original: be
Revised: be

Original: swapped
Revised: swapped

Original: during the metalearning
Revised: during the metalearning

Original: optimization
Revised: optimization

Original: process.
Revised: process.

Original: Moreover, we have
Revised: Moreover, we have

Original: added two
Revised: added two

Original: conditions
Revised: conditions

Original: that
Revised: that

Original: allow us to externally tune the learning rate as a function of time and to provide symmetric or positive clamping of the
Revised: allow us to externally tune the learning rate as a function of time and to provide symmetric or positive clamping of the

Original: synaptic
Revised: synaptic

Original: weights in order to prevent
Revised: weights in order to prevent

Original: instabilities
Revised: instabilities

Original: in the algorithms.
Revised: in the algorithms.

Original: 2.3 Mixed-Integer Optimization Framework
Revised: 2.3 Mixed-Integer Optimization Framework

Original: The parameter space in the proposed multilayer neuromodulated architecture is composed of categorical variables (e.g., the selection of the local learning rule), integer parameters (e.g., the dimension of hidden layer), and continuous parameters in each of the learning rules.
Revised: The parameter space in the proposed multilayer neuromodulated architecture is composed of categorical variables (e.g., the selection of the local learning rule), integer parameters (e.g., the dimension of hidden layer), and continuous parameters in each of the learning rules.

Original: We adopt a parallel asynchronous-model-based search approach (AMBS) (Balaprakash et al., 2018) to ﬁnd the high-performing parameter conﬁguration in this mixed (categorical, continuous, integer) search space.
Revised: We adopt a parallel asynchronous-model-based search approach (AMBS) (Balaprakash et al., 2018) to ﬁnd the high-performing parameter conﬁguration in this mixed (categorical, continuous, integer) search space.

Original: Our AMBS approach consists of sampling a number of parameter conﬁgurations and progressively ﬁtting a surrogate model over the parameter conﬁgurations’ accuracy metric space.
Revised: Our AMBS approach consists of sampling a number of parameter conﬁgurations and progressively ﬁtting a surrogate model over the parameter conﬁgurations’ accuracy metric space.

Original: This surrogate model is asynchronously updated as new conﬁgurations are being evaluated by the parallel processes, which are then used to obtain conﬁgurations that will be evaluated in the next iteration.
Revised: This surrogate model is asynchronously updated as new conﬁgurations are being evaluated by the parallel processes, which are then used to obtain conﬁgurations that will be evaluated in the next iteration.

Original: Crucial to the optimization approach is the choice of the surrogate model, since this model generates the conﬁguration to evaluate in the mixed search space.
Revised: Crucial to the optimization approach is the choice of the surrogate model, since this model generates the conﬁguration to evaluate in the mixed search space.

Original: The AMBS adopts random forest approach to build efﬁcient regression models on this search space.
Revised: The AMBS adopts random forest approach to build efﬁcient regression models on this search space.

Original: The random forest is an ensemble learning approach that builds multiple decision trees and uses bootstrap aggregation (or bagging) to combine them to produce a model with better predictive accuracy and lower variance.
Revised: The random forest is an ensemble learning approach that builds multiple decision trees and uses bootstrap aggregation (or bagging) to combine them to produce a model with better predictive accuracy and lower variance.

Original: Another key choice for the AMBS approach is the acquisition function, which encapsulates criteria to choose the most promising conﬁgurations to evaluate next.
Revised: Another key choice for the AMBS approach is the acquisition function, which encapsulates criteria to choose the most promising conﬁgurations to evaluate next.

Original: Hence, the acquisition function is key to maintaining the explorationexploitation balance during the search.
Revised: Hence, the acquisition function is key to maintaining the explorationexploitation balance during the search.

Original: The AMBS adopts the lower conﬁdence bound (LCB) acquisition function.
Revised: The AMBS adopts the lower conﬁdence bound (LCB) acquisition function.

Original: We set the kappa to a large value, which increases exploration.
Revised: We set the kappa to a large value, which increases exploration.

Original: We do so to accommodate the high variability we observed in the accuracy metrics within and across the local learning rules that led them to local minimum when a smaller value of kappa was used.
Revised: We do so to accommodate the high variability we observed in the accuracy metrics within and across the local learning rules that led them to local minimum when a smaller value of kappa was used.

Original:  Online and Continual Learning Experiments
Revised:  Online and Continual Learning Experiments

Original: We focus on a speciﬁc type of experiment in which the system is subject to a stream of data and labels during a predetermined number of epochs.
Revised: We focus on a speciﬁc type of experiment in which the system is subject to a stream of data and labels during a predetermined number of epochs.

Original: This constitutes a single episode.
Revised: This constitutes a single episode.

Original: Through the synaptic plasticity mechanisms implemented in the network, the weights evolve during the episode.
Revised: Through the synaptic plasticity mechanisms implemented in the network, the weights evolve during the episode.

Original: At the end of the episode, the system is evaluated against the testing dataset to validate its accuracy.
Revised: At the end of the episode, the system is evaluated against the testing dataset to validate its accuracy.

Original: By concatenating multiple episodes involving different tasks and datasets, we can create a curriculum to evaluate the system’s ability to carry out continual learning.
Revised: By concatenating multiple episodes involving different tasks and datasets, we can create a curriculum to evaluate the system’s ability to carry out continual learning.

Original: The resulting accuracy is the metric that the optimization framework uses to carry out the exploration of the architecture and hyperparameter space and ﬁnd the optimal conﬁgurations.
Revised: The resulting accuracy is the metric that the optimization framework uses to carry out the exploration of the architecture and hyperparameter space and ﬁnd the optimal conﬁgurations.

Original: As part of the optimization process, multiple episodes are run for systems with different architectures and speciﬁc hyperparameters.
Revised: As part of the optimization process, multiple episodes are run for systems with different architectures and speciﬁc hyperparameters.

Original: In all these cases, the system starts from identical starting conditions, so that no knowledge is transferred between episodes that do not belong to the same curriculum.
Revised: In all these cases, the system starts from identical starting conditions, so that no knowledge is transferred between episodes that do not belong to the same curriculum.

Original: We consider two different learning modalities to evaluate the proposed approach:
Revised: We consider two different learning modalities to evaluate the proposed approach:

Original: (a) Non-Incremental Learning: There is a single episode in the curriculum, which has access to all the data and hence performs a single task (e.g., multiclass classiﬁcation); (b) Class-Incremental Learning: The synaptic weights are updated incrementally, where a speciﬁc task (e
Revised: (a) Non-Incremental Learning: There is a single episode in the curriculum, which has access to all the data and hence performs a single task (e.g., multiclass classiﬁcation); (b) Class-Incremental Learning: The synaptic weights are updated incrementally, where a speciﬁc task (e

Original: g., two-
Revised: g., two-

Original: class classiﬁcation) is performed in each episode using data streams from distinct (clearly separated) classes.
Revised: class classiﬁcation) is performed in each episode using data streams from distinct (clearly separated) classes.

Original: The curriculum involves a concatenation of multiple episodes such that the ﬁnal accuracy measures the model’s ability to predict on test data from all the classes, spanning across episodes.
Revised: The curriculum involves a concatenation of multiple episodes such that the ﬁnal accuracy measures the model’s ability to predict on test data from all the classes, spanning across episodes.

Original: By changing tasks, datasets, and learning modalities, we build a collection of high-performing conﬁgurations for each type of experiment.
Revised: By changing tasks, datasets, and learning modalities, we build a collection of high-performing conﬁgurations for each type of experiment.

Original: We use these conﬁgurations to explore the transferability of optimal learning conditions to other datasets and to different types of tasks.
Revised: We use these conﬁgurations to explore the transferability of optimal learning conditions to other datasets and to different types of tasks.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: We ﬁrst describe the results obtained in the non-incremental learning scenario, followed by incremental learning scenario.
Revised: We ﬁrst describe the results obtained in the non-incremental learning scenario, followed by incremental learning scenario.

Original: For both scenarios, we demonstrate that the accuracies we obtain are on par with (or outperformed) other state-of-the-art approaches without explicit memory consideration.
Revised: For both scenarios, we demonstrate that the accuracies we obtain are on par with (or outperformed) other state-of-the-art approaches without explicit memory consideration.

Original: Finally, we perform experiments exploring the transferability of optimal learning conditions to tasks and datasets and ﬁnd that it is strongly correlated to similarity between the datasets.
Revised: Finally, we perform experiments exploring the transferability of optimal learning conditions to tasks and datasets and ﬁnd that it is strongly correlated to similarity between the datasets.

Original: 4.1 Non-Incremental Learning
Revised: 4.1 Non-Incremental Learning

Original: In this experiment, we demonstrate the learning capability of our multilayer neuromodulated learning framework on single episode curriculum learning task, speciﬁcally multi-class classiﬁcation.
Revised: In this experiment, we demonstrate the learning capability of our multilayer neuromodulated learning framework on single episode curriculum learning task, speciﬁcally multi-class classiﬁcation.

Original: We consider MNIST, Fashion MNIST (Xiao et al., 2017b)
Revised: We consider MNIST, Fashion MNIST (Xiao et al., 2017b)

Original: (F-MNIST), and Extended MNIST (Cohen et al., 2017)
Revised: (F-MNIST), and Extended MNIST (Cohen et al., 2017)

Original: (E-MNIST) datasets because of the homogeneity in image and class sizes and the existence of benchmarks in continual learning.
Revised: (E-MNIST) datasets because of the homogeneity in image and class sizes and the existence of benchmarks in continual learning.

Original: For each dataset, we jointly optimize over the local learning rules and their parameters to ﬁnd the optimal conﬁguration.
Revised: For each dataset, we jointly optimize over the local learning rules and their parameters to ﬁnd the optimal conﬁguration.

Original: We employed SPARSE as the feature representation layer and three supervised learning rules ( MSE , GEN , OJA ) for label prediction.
Revised: We employed SPARSE as the feature representation layer and three supervised learning rules ( MSE , GEN , OJA ) for label prediction.

Original: While the number of parameters in each of the learning rule differs, the search space is deﬁned to exploit the common parameters, whose dimensionality is deﬁned by the rule with maximum parameters.
Revised: While the number of parameters in each of the learning rule differs, the search space is deﬁned to exploit the common parameters, whose dimensionality is deﬁned by the rule with maximum parameters.

Original: In addition, only the subset of parameters present in the local-learning rule are active at search time.
Revised: In addition, only the subset of parameters present in the local-learning rule are active at search time.

Original: For example, the parameter β 3 l deﬁnes the weight clamping and is common to the three rules; the GEN rule has all the parameters ( β 1 l , β 2 l , β 3 l ) active; whereas MSE has only β 3 l active, so the other parameters ( β 1 l , β 2 l ) are set to None while exploring the search space.
Revised: For example, the parameter β 3 l deﬁnes the weight clamping and is common to the three rules; the GEN rule has all the parameters ( β 1 l , β 2 l , β 3 l ) active; whereas MSE has only β 3 l active, so the other parameters ( β 1 l , β 2 l ) are set to None while exploring the search space.

Original: For each conﬁguration evaluated during the optimization, the model is run for 0 .
Revised: For each conﬁguration evaluated during the optimization, the model is run for 0 .

Original: 5 epochs.
Revised: 5 epochs.

Original: The best accuracy obtained and the corresponding optimal parameters for all the datasets are shown in Table 1.
Revised: The best accuracy obtained and the corresponding optimal parameters for all the datasets are shown in Table 1.

Original: The search trajectory showing the learning rule and test accuracy of the conﬁgurations evaluated as a function of time is shown Fig.
Revised: The search trajectory showing the learning rule and test accuracy of the conﬁgurations evaluated as a function of time is shown Fig.

Original: 2 for MNIST and F-MNIST.
Revised: 2 for MNIST and F-MNIST.

Original: In both the cases, the initial conﬁgurations range across the different learning rules, but the algorithm quickly ﬁnds the potential learning rule and evaluates more conﬁgurations from it.
Revised: In both the cases, the initial conﬁgurations range across the different learning rules, but the algorithm quickly ﬁnds the potential learning rule and evaluates more conﬁgurations from it.

Original: The corresponding accuracies for each of the three datasets obtaining by using the optimal conﬁgurations, but run of 4 epochs is shown in Table 2.
Revised: The corresponding accuracies for each of the three datasets obtaining by using the optimal conﬁgurations, but run of 4 epochs is shown in Table 2.

Original: These accuracies are on par with (or outperform) other shallow network architectures (Yanguas-Gil et al., 2019; Xiao et al., 2017a;
Revised: These accuracies are on par with (or outperform) other shallow network architectures (Yanguas-Gil et al., 2019; Xiao et al., 2017a;

Original: Cohen et al., 2017).
Revised: Cohen et al., 2017).

Original: [Figure]
Revised: [Figure]

Original: 4.2 Incremental Learning
Revised: 4.2 Incremental Learning

Original: In this experiment, we demonstrate the learning capability of our approach in the class-incremental learning scenario, where multiple episodes are present per learning curriculum.
Revised: In this experiment, we demonstrate the learning capability of our approach in the class-incremental learning scenario, where multiple episodes are present per learning curriculum.

Original: We evaluate our method on the Split-MNIST (Farquhar &
Revised: We evaluate our method on the Split-MNIST (Farquhar &

Original: Gal, 2018) and Permuted-MNIST (Goodfellow et al., 2013) incremental learning benchmarks that have been extensively adopted in the literature (Shin et al., 2017; Zenke et al., 2017; Nguyen et al., 2017).
Revised: Gal, 2018) and Permuted-MNIST (Goodfellow et al., 2013) incremental learning benchmarks that have been extensively adopted in the literature (Shin et al., 2017; Zenke et al., 2017; Nguyen et al., 2017).

Original: The Split-MNIST data is prepared by splitting the original MNIST dataset (both training and testing splits) consisting of ten digits into ﬁve two-class classiﬁcation tasks.
Revised: The Split-MNIST data is prepared by splitting the original MNIST dataset (both training and testing splits) consisting of ten digits into ﬁve two-class classiﬁcation tasks.

Original: This deﬁnes an incremental learning sce-
Revised: This deﬁnes an incremental learning sce-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: nario in which the model sees these ﬁve tasks incrementally one after the other.
Revised: nario in which the model sees these ﬁve tasks incrementally one after the other.

Original: The Permuted-MNIST is also prepared from the original MNIST data by permuting the pixels in the images.
Revised: The Permuted-MNIST is also prepared from the original MNIST data by permuting the pixels in the images.

Original: A unique permutation is applied to all the images in order to generate a new ten-class classiﬁcation task.
Revised: A unique permutation is applied to all the images in order to generate a new ten-class classiﬁcation task.

Original: The number of distinct permutations applied represents the length of the task sequence.
Revised: The number of distinct permutations applied represents the length of the task sequence.

Original: For instance, we adopted a ten-permuted MNIST, which produces ten (ten-class classiﬁcation) tasks that are sequentially seen by the model.
Revised: For instance, we adopted a ten-permuted MNIST, which produces ten (ten-class classiﬁcation) tasks that are sequentially seen by the model.

Original: The ﬁrst task is the original MNIST data, and the subsequent nine tasks are obtained by permuting the original data.
Revised: The ﬁrst task is the original MNIST data, and the subsequent nine tasks are obtained by permuting the original data.

Original: Following (Hsu et al., 2018), for Split-MNIST, a simple two-layer multilayered perceptron (MLP) neural network with 400 neurons per layer is adopted to evaluate the baseline accuracy, as well as the accuracy with other incremental learning algorithms.
Revised: Following (Hsu et al., 2018), for Split-MNIST, a simple two-layer multilayered perceptron (MLP) neural network with 400 neurons per layer is adopted to evaluate the baseline accuracy, as well as the accuracy with other incremental learning algorithms.

Original: Similarly, for Permuted-MNIST, we use two-layer MLP with 1,000 neurons per layer to evaluate the accuracy of the baseline and other incremental learning algorithms.
Revised: Similarly, for Permuted-MNIST, we use two-layer MLP with 1,000 neurons per layer to evaluate the accuracy of the baseline and other incremental learning algorithms.

Original: See (Hsu et al., 2018) for more details on the hyperparameters used for both these models.
Revised: See (Hsu et al., 2018) for more details on the hyperparameters used for both these models.

Original: The ﬁrst baseline is a naive approach in which the MLP network (with
Revised: The ﬁrst baseline is a naive approach in which the MLP network (with

Original: Adagrad optimizer) is trained by progressively updating the parameters through backpropagation as new tasks are observed.
Revised: Adagrad optimizer) is trained by progressively updating the parameters through backpropagation as new tasks are observed.

Original: The second baseline is a naive rehearsal (experience replay) approach that stores a fraction of data from previous tasks randomly and uses that while training incrementally.
Revised: The second baseline is a naive rehearsal (experience replay) approach that stores a fraction of data from previous tasks randomly and uses that while training incrementally.

Original: Among the popular incremental learning approaches, we choose representative ones from regularization-based (Online EWC (Schwarz et al., 2018), SI (Zenke et al., 2017),
Revised: Among the popular incremental learning approaches, we choose representative ones from regularization-based (Online EWC (Schwarz et al., 2018), SI (Zenke et al., 2017),

Original: MAS (Aljundi et al., 2018)), and memory-based methods (GEM (Lopez-Paz & Ranzato, 2017), DGR (Shin et al., 2017), Rtf (van de Ven & Tolias, 2018)).
Revised: MAS (Aljundi et al., 2018)), and memory-based methods (GEM (Lopez-Paz & Ranzato, 2017), DGR (Shin et al., 2017), Rtf (van de Ven & Tolias, 2018)).

Original: We also compare the non-incremental learning scenario with the corresponding baseline models for both datasets as well as our multilayer neuromodulated learning approach.
Revised: We also compare the non-incremental learning scenario with the corresponding baseline models for both datasets as well as our multilayer neuromodulated learning approach.

Original: The results of the incremental learning experiments are shown in Table 3.
Revised: The results of the incremental learning experiments are shown in Table 3.

Original: For the non-incremental learning case, where the data from all tasks are provided at the same time, the baseline MLP model gives an accuracy of 97 .
Revised: For the non-incremental learning case, where the data from all tasks are provided at the same time, the baseline MLP model gives an accuracy of 97 .

Original: 53% and 97 .
Revised: 53% and 97 .

Original: 95% for the
Revised: 95% for the

Original: Split-MNIST and Permuted-MNIST datasets, respectively, after 4 epochs of training for the former and 10 epochs for the latter, following (Hsu et al., 2018).
Revised: Split-MNIST and Permuted-MNIST datasets, respectively, after 4 epochs of training for the former and 10 epochs for the latter, following (Hsu et al., 2018).

Original: We use the best-performing parameter conﬁguration for MNIST data obtained for the non-incremental learning (Table 1) to evaluate the performance of our architecture in these two datasets.
Revised: We use the best-performing parameter conﬁguration for MNIST data obtained for the non-incremental learning (Table 1) to evaluate the performance of our architecture in these two datasets.

Original: This transfer of parameter conﬁguration from the non-incremental to the incremental learning scenario is further discussed in the next section.
Revised: This transfer of parameter conﬁguration from the non-incremental to the incremental learning scenario is further discussed in the next section.

Original: With this parameter conﬁguration, we found that the accuracy of our model is 97 .
Revised: With this parameter conﬁguration, we found that the accuracy of our model is 97 .

Original: 45 on Split-MNIST trained for 4 epochs and 96 .
Revised: 45 on Split-MNIST trained for 4 epochs and 96 .

Original: 69% on the Permuted-MNIST trained for just 1 epoch.
Revised: 69% on the Permuted-MNIST trained for just 1 epoch.

Original: These values are very close to the MLP model accuracy— 97 .on
Revised: These values are very close to the MLP model accuracy— 97 .on

Original: Split-MNIST trained for 4 epochs and 97 .
Revised: Split-MNIST trained for 4 epochs and 97 .

Original: 95% on the
Revised: 95% on the

Original: Permuted-MNIST trained for 10 epochs .
Revised: Permuted-MNIST trained for 10 epochs .

Original: For the class-incremental learning on Split-MNIST and the
Revised: For the class-incremental learning on Split-MNIST and the

Original: Permuted-MNIST, our model (using MNIST parameter conﬁguration from Table 1) signiﬁcantly outperforms both the baseline and other non-memory-based incremental learning approaches considered.
Revised: Permuted-MNIST, our model (using MNIST parameter conﬁguration from Table 1) signiﬁcantly outperforms both the baseline and other non-memory-based incremental learning approaches considered.

Original: In addition, we obtain accuracy comparable to the state-of-the-art memory-based models for Split-MNIST data and outperform all the other incremental learning algorithms on the Permuted-MNIST data with only three epochs (as compared with ten epochs for all the other algorithms).
Revised: In addition, we obtain accuracy comparable to the state-of-the-art memory-based models for Split-MNIST data and outperform all the other incremental learning algorithms on the Permuted-MNIST data with only three epochs (as compared with ten epochs for all the other algorithms).

Original: The naive rehearsal approach outperforms all the other approaches, but it comes with the additional memory overhead.
Revised: The naive rehearsal approach outperforms all the other approaches, but it comes with the additional memory overhead.

Original: 4.3 Transferability Study
Revised: 4.3 Transferability Study

Original: Identiﬁcation of the best conﬁguration for each group of classes (a dataset) is based on the assumption that all the data is available at the beginning of the training procedure.
Revised: Identiﬁcation of the best conﬁguration for each group of classes (a dataset) is based on the assumption that all the data is available at the beginning of the training procedure.

Original: However, for many online learning scenarios, both incremental and non-incremental, this conﬁguration might not be known a priori .
Revised: However, for many online learning scenarios, both incremental and non-incremental, this conﬁguration might not be known a priori .

Original: This raises the question of transfer metalearning: how to effectively optimize a learning algorithm to learn unknown tasks and data.
Revised: This raises the question of transfer metalearning: how to effectively optimize a learning algorithm to learn unknown tasks and data.

Original: To address this problem, we empirically study the effect of transferring optimal conﬁgurations tasks for the incremental learning scenario, and
Revised: To address this problem, we empirically study the effect of transferring optimal conﬁgurations tasks for the incremental learning scenario, and

Original: [Table]
Revised: [Table]

Original: across datasets for the non-incremental learning case.
Revised: across datasets for the non-incremental learning case.

Original: Across tasks: The best conﬁguration obtained through the joint optimization on each dataset separately (Table 1) is used to evaluate the incremental learning accuracy with
Revised: Across tasks: The best conﬁguration obtained through the joint optimization on each dataset separately (Table 1) is used to evaluate the incremental learning accuracy with

Original: Split-MNIST and Permuted-MNIST datasets.
Revised: Split-MNIST and Permuted-MNIST datasets.

Original: The results using the conﬁgurations corresponding to MNIST, F-MNIST, and E-MNIST are shown in Table 4.
Revised: The results using the conﬁgurations corresponding to MNIST, F-MNIST, and E-MNIST are shown in Table 4.

Original: We observe that the conﬁguration learned on the MNIST dataset is readily transferable to incremental learning on Split-MNIST and the
Revised: We observe that the conﬁguration learned on the MNIST dataset is readily transferable to incremental learning on Split-MNIST and the

Original: Permuted-MNIST data, as seen in the preceding section
Revised: Permuted-MNIST data, as seen in the preceding section

Original: [Table]
Revised: [Table]

Original: The conﬁguration learned with the E-MNIST data leads to an accuracy difference of only 0 .
Revised: The conﬁguration learned with the E-MNIST data leads to an accuracy difference of only 0 .

Original: 71% and 0 .
Revised: 71% and 0 .

Original: 50% , respectively, on these incremental learning datasets.
Revised: 50% , respectively, on these incremental learning datasets.

Original: The F-MNIST conﬁguration, on the other hand, leads to an accuracy drop of 0 .
Revised: The F-MNIST conﬁguration, on the other hand, leads to an accuracy drop of 0 .

Original: 98% and 4 .
Revised: 98% and 4 .

Original: 14% , respectively on them.
Revised: 14% , respectively on them.

Original: This observation suggests that the transfer learning conﬁguration (both the local learning rule and its corresponding parameters) to the incremental learning experiments needs to be done carefully, which otherwise would lead to suboptimal accuracy.
Revised: This observation suggests that the transfer learning conﬁguration (both the local learning rule and its corresponding parameters) to the incremental learning experiments needs to be done carefully, which otherwise would lead to suboptimal accuracy.

Original: Across datasets: We seek to understand and characterize the dependence of dataset similarity (deﬁned as a distance metric) and the transferability of optimal conﬁgurations learned on a dataset to other (different) datasets.
Revised: Across datasets: We seek to understand and characterize the dependence of dataset similarity (deﬁned as a distance metric) and the transferability of optimal conﬁgurations learned on a dataset to other (different) datasets.

Original: To this end, we choose the best conﬁguration obtained through the joint optimization on each dataset (Table 1) and use that for transfer learning and evaluating the accuracy for the remaining datasets.
Revised: To this end, we choose the best conﬁguration obtained through the joint optimization on each dataset (Table 1) and use that for transfer learning and evaluating the accuracy for the remaining datasets.

Original: The results for transfer learning across
Revised: The results for transfer learning across

Original: MNIST, F-MNIST, and E-MNIST are summarized in Table 4, where a column represents the accuracy on a particular dataset obtained by transfer learning through the optimal conﬁgurations learned on standalone MNIST, F-MNIST, and E-MNIST datasets.
Revised: MNIST, F-MNIST, and E-MNIST are summarized in Table 4, where a column represents the accuracy on a particular dataset obtained by transfer learning through the optimal conﬁgurations learned on standalone MNIST, F-MNIST, and E-MNIST datasets.

Original: The diagonal elements are the same as the accuracy values obtained previously for four epoch runs with the same data-optimal conﬁguration combination (Table 2).
Revised: The diagonal elements are the same as the accuracy values obtained previously for four epoch runs with the same data-optimal conﬁguration combination (Table 2).

Original: We ﬁnd that the conﬁgurations learned on MNIST and E-MNIST are readily exchangeable without signiﬁcant loss in accuracy.
Revised: We ﬁnd that the conﬁgurations learned on MNIST and E-MNIST are readily exchangeable without signiﬁcant loss in accuracy.

Original: These conﬁgurations, however, do not transfer well to the F-MNIST dataset, as seen by the accuracy decrease of 11 .
Revised: These conﬁgurations, however, do not transfer well to the F-MNIST dataset, as seen by the accuracy decrease of 11 .

Original: 45% and 11 .
Revised: 45% and 11 .

Original: 12% , respectively, with
Revised: 12% , respectively, with

Original: MNIST and E-MNIST.
Revised: MNIST and E-MNIST.

Original: On the other hand, the F-MNIST conﬁgurations also lead to a decrease in accuracy of 5 .
Revised: On the other hand, the F-MNIST conﬁgurations also lead to a decrease in accuracy of 5 .

Original: 53% and 3 .
Revised: 53% and 3 .

Original: 75% , respectively, when transferred to MNIST and
Revised: 75% , respectively, when transferred to MNIST and

Original: E-MNIST data.
Revised: E-MNIST data.

Original: Distance metrics as a measure of transferability: To rationalize the results shown in Table 4, we have explored the correlation between the drop in performance between datasets and their effective dimensionality and the distance between datasets.
Revised: Distance metrics as a measure of transferability: To rationalize the results shown in Table 4, we have explored the correlation between the drop in performance between datasets and their effective dimensionality and the distance between datasets.

Original: We focus on a deﬁnition of dimensionality derived from the covariance matrix of the complete dataset.
Revised: We focus on a deﬁnition of dimensionality derived from the covariance matrix of the complete dataset.

Original: The eigenvalues of the covariance matrix yields the principal components of the dataset.
Revised: The eigenvalues of the covariance matrix yields the principal components of the dataset.

Original: We consider as a metric of dimensionality the sum of squares of the eigenvalues ( λ ), so that
Revised: We consider as a metric of dimensionality the sum of squares of the eigenvalues ( λ ), so that

Original: [Equation]
Revised: [Equation]

Original: This metric has been used in the past to characterize sparse representations (Litwin-Kumar et al., 2017).
Revised: This metric has been used in the past to characterize sparse representations (Litwin-Kumar et al., 2017).

Original: The interpretation of this metric is that when the variance is equally distributed across all dimensions, Eq.
Revised: The interpretation of this metric is that when the variance is equally distributed across all dimensions, Eq.

Original: 6 yields a dimension d = N , where N is the number of dimensions in the space (784 in this case).
Revised: 6 yields a dimension d = N , where N is the number of dimensions in the space (784 in this case).

Original: If all the variance is concentrated in a single eigenvalue, then d = 1 .
Revised: If all the variance is concentrated in a single eigenvalue, then d = 1 .

Original: To quantify the distance and separability between categories in the dataset, we have used the cosine distance metric.
Revised: To quantify the distance and separability between categories in the dataset, we have used the cosine distance metric.

Original: In
Revised: In

Original: Table 6 we have applied the traditional deﬁnition of distance between two clusters and calculated the minimum distance between any two categories of two datasets.
Revised: Table 6 we have applied the traditional deﬁnition of distance between two clusters and calculated the minimum distance between any two categories of two datasets.

Original: We also show the maximum separation for comparison.
Revised: We also show the maximum separation for comparison.

Original: When these metrics are used within a given dataset, we obtain the minimum and maximum distance between two categories of
Revised: When these metrics are used within a given dataset, we obtain the minimum and maximum distance between two categories of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: the same dataset, which could be interpreted as a measure of separability.
Revised: the same dataset, which could be interpreted as a measure of separability.

Original: Figure 3 shows the drop in classiﬁcation accuracy during transfer metalearning as a function of a transfer coefﬁcient obtained from two different metrics: the relative difference in the eigenvalue dimension given by Eq.
Revised: Figure 3 shows the drop in classiﬁcation accuracy during transfer metalearning as a function of a transfer coefﬁcient obtained from two different metrics: the relative difference in the eigenvalue dimension given by Eq.

Original: 6 and the minimum cosine distance between the two datasets.
Revised: 6 and the minimum cosine distance between the two datasets.

Original: Both values are normalized as
Revised: Both values are normalized as

Original: [Equation]
Revised: [Equation]

Original: where M 1 is the metric of the dataset selected to run the experiment and M 2 is the metric of the dataset whose optimal conﬁguration is used.
Revised: where M 1 is the metric of the dataset selected to run the experiment and M 2 is the metric of the dataset whose optimal conﬁguration is used.

Original: Note that this transfer coefﬁcient is not symmetric because of the different normalization in Eq.
Revised: Note that this transfer coefﬁcient is not symmetric because of the different normalization in Eq.

Original: 7, as should be expected since transfer metalearning is directional.
Revised: 7, as should be expected since transfer metalearning is directional.

Original: The results clearly show that as the distance between the datasets increases, transfer learning the conﬁgurations across them leads to a decrease in accuracy.
Revised: The results clearly show that as the distance between the datasets increases, transfer learning the conﬁgurations across them leads to a decrease in accuracy.

Original:  Related Work
Revised:  Related Work

Original: Incremental learning, also referred to as continual or lifelong learning (Thrun & Pratt, 2012), describes a learning modality in which a model seeks to learn from data and tasks that are sequentially presented to it.
Revised: Incremental learning, also referred to as continual or lifelong learning (Thrun & Pratt, 2012), describes a learning modality in which a model seeks to learn from data and tasks that are sequentially presented to it.

Original: Several incremental learning approaches have been presented in the literature, which can loosely be categorized into three classes: (1) novel neural architectures or customization of the common ones; (2) regularization strategies that impose constraints to boost knowledge retainment; and (3) metalearning, which uses a series of tasks to learn a common parameter conﬁguration that is easily adaptable for new tasks.
Revised: Several incremental learning approaches have been presented in the literature, which can loosely be categorized into three classes: (1) novel neural architectures or customization of the common ones; (2) regularization strategies that impose constraints to boost knowledge retainment; and (3) metalearning, which uses a series of tasks to learn a common parameter conﬁguration that is easily adaptable for new tasks.

Original: Algorithms in the ﬁrst category include bio-inspired dual-memory architecture (Parisi et al., 2019); progressive neural networks (Rusu et al., 2016) that explicitly support information transfer across sequences of tasks through network expansion; and deep generative replay (Shin et al., 2017), which proposed a cooperative dual model architecture framework, inspired by hippocampus, that retains past knowledge by the concurrent replay of generated pseudodata.
Revised: Algorithms in the ﬁrst category include bio-inspired dual-memory architecture (Parisi et al., 2019); progressive neural networks (Rusu et al., 2016) that explicitly support information transfer across sequences of tasks through network expansion; and deep generative replay (Shin et al., 2017), which proposed a cooperative dual model architecture framework, inspired by hippocampus, that retains past knowledge by the concurrent replay of generated pseudodata.

Original: The second category consists of algorithms such as elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) that computes synaptic importance using a Fisher importance matrix-based regularization; synaptic intelligence (SI) (Zenke et al., 2017), whose regularization penalty is similar to EWC but is computed online at per-synapse level; and learning without forgetting (Li & Hoiem, 2017), which applies a distillation loss on the attention-enabled deep networks seeking to minimize task overlap.
Revised: The second category consists of algorithms such as elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) that computes synaptic importance using a Fisher importance matrix-based regularization; synaptic intelligence (SI) (Zenke et al., 2017), whose regularization penalty is similar to EWC but is computed online at per-synapse level; and learning without forgetting (Li & Hoiem, 2017), which applies a distillation loss on the attention-enabled deep networks seeking to minimize task overlap.

Original: The third category consists of algorithms such as online metalearning (Javed & White, 2019), neuromodulated metalearning algorithm (Beaulieu et al., 2020), and incremental task-agnostic metalearning (Rajasegaran et al., 2020) that show great promise but are not particularly suited for the memory-constrained continual learning scenarios that could be relevant in edge computing.
Revised: The third category consists of algorithms such as online metalearning (Javed & White, 2019), neuromodulated metalearning algorithm (Beaulieu et al., 2020), and incremental task-agnostic metalearning (Rajasegaran et al., 2020) that show great promise but are not particularly suited for the memory-constrained continual learning scenarios that could be relevant in edge computing.

Original: These algorithms are characterized by large network sizes and memory buffer requirements and have a few restricting assumptions on the structure of the data; hence we do not include a comparison with this class of approaches.
Revised: These algorithms are characterized by large network sizes and memory buffer requirements and have a few restricting assumptions on the structure of the data; hence we do not include a comparison with this class of approaches.

Original: The multilayer neuromodulated architecture approach we proposed falls under the ﬁrst class of continual learning methods discussed earlier.
Revised: The multilayer neuromodulated architecture approach we proposed falls under the ﬁrst class of continual learning methods discussed earlier.

Original:  Conclusions
Revised:  Conclusions

Original: We developed a multilayer, recurrent neuromorphic architecture capable of online continual learning in a memoryconstrained setting, where large models and data storage/replay is limited.
Revised: We developed a multilayer, recurrent neuromorphic architecture capable of online continual learning in a memoryconstrained setting, where large models and data storage/replay is limited.

Original: The proposed architecture consists of a processing component and a learning component, where learning can be viewed as a dynamic process that alters the architecture itself via recurrent interactions as it processes and assigns valence to certain inputs and create associations over time.
Revised: The proposed architecture consists of a processing component and a learning component, where learning can be viewed as a dynamic process that alters the architecture itself via recurrent interactions as it processes and assigns valence to certain inputs and create associations over time.

Original: Our approach parameterizes multiple synaptic plasticity mechanisms as layers of this network sharing a common interface.
Revised: Our approach parameterizes multiple synaptic plasticity mechanisms as layers of this network sharing a common interface.

Original: This allows us to cast the optimization of the architecture’s learning capabilities as an optimization problem and to employ a Bayesian optimization-based search to ﬁnd optimal task-speciﬁc conﬁgurations in the mixed (categorical, continuous) integer space that spans over the choice of the learning algorithms, their speciﬁc hyperparameters, and feature extraction layers.
Revised: This allows us to cast the optimization of the architecture’s learning capabilities as an optimization problem and to employ a Bayesian optimization-based search to ﬁnd optimal task-speciﬁc conﬁgurations in the mixed (categorical, continuous) integer space that spans over the choice of the learning algorithms, their speciﬁc hyperparameters, and feature extraction layers.

Original: We demonstrate our approach using two different learning scenarios.
Revised: We demonstrate our approach using two different learning scenarios.

Original: The ﬁrst is a non-incremental learning scenario, where the learning curriculum consists of a episode that has access to all tasks, and the model is updated online as the data streams in.
Revised: The ﬁrst is a non-incremental learning scenario, where the learning curriculum consists of a episode that has access to all tasks, and the model is updated online as the data streams in.

Original: The second is a class-incremental learning scenario, where data from each task is presented sequentially such that each episode in the learning curriculum consists of a unique task.
Revised: The second is a class-incremental learning scenario, where data from each task is presented sequentially such that each episode in the learning curriculum consists of a unique task.

Original: The model is then updated online in each episode and continually with a sequence of these episodes.
Revised: The model is then updated online in each episode and continually with a sequence of these episodes.

Original: In the non-incremental learning case, our algorithm identiﬁed conﬁgurations capable of obtaining online learning accuracies of 97 .
Revised: In the non-incremental learning case, our algorithm identiﬁed conﬁgurations capable of obtaining online learning accuracies of 97 .

Original: 45 for MNIST, 94 .
Revised: 45 for MNIST, 94 .

Original: 32 for
Revised: 32 for

Original: Fashion MNIST, and 97 .
Revised: Fashion MNIST, and 97 .

Original: 34 for Extended MNIST, which were on par with (or outperformed) the performance of static shallow neural networks run for four epochs.
Revised: 34 for Extended MNIST, which were on par with (or outperformed) the performance of static shallow neural networks run for four epochs.

Original: The optimal conﬁgurations in our architecture were obtained by imposing a limit of 0 .
Revised: The optimal conﬁgurations in our architecture were obtained by imposing a limit of 0 .

Original: 5 epochs to learn each dataset, which was cheap computationally and led to good conﬁgurations.
Revised: 5 epochs to learn each dataset, which was cheap computationally and led to good conﬁgurations.

Original: In the class-incremental learning case, we obtained an accuracy of 92 .
Revised: In the class-incremental learning case, we obtained an accuracy of 92 .

Original: 76 and 96 .
Revised: 76 and 96 .

Original: 62 on the Split-MNIST and
Revised: 62 on the Split-MNIST and

Original: Permuted-MNIST data, which consisted of ﬁve and ten sequential tasks, respectively.
Revised: Permuted-MNIST data, which consisted of ﬁve and ten sequential tasks, respectively.

Original: Hence, our approach clearly outperformed the memory-free approaches, whose accuracies obtained a maximum of 20 . 88 and 85 . 57 , respectively.
Revised: Hence, our approach clearly outperformed the memory-free approaches, whose accuracies obtained a maximum of 20 . 88 and 85 . 57 , respectively.

Original: The memory replay-based continual learning approaches gave a maximum accuracy of 92 .
Revised: The memory replay-based continual learning approaches gave a maximum accuracy of 92 .

Original: 56 with Rtf and 96 .
Revised: 56 with Rtf and 96 .

Original: 72 with
Revised: 72 with

Original: GEM using four and ten epoch of training, respectively, on the two datasets, whereas our approach produced an accuracy of 92.76 with four epochs on Split-MNIST and 96.
Revised: GEM using four and ten epoch of training, respectively, on the two datasets, whereas our approach produced an accuracy of 92.76 with four epochs on Split-MNIST and 96.

Original: with one epoch on Permuted-MNIST.
Revised: with one epoch on Permuted-MNIST.

Original: These results demonstrate that memoryless approaches such as the one proposed here can achieve performances on par with the memoryreplay-based approaches used as benchmarks, without the additional memory overhead.
Revised: These results demonstrate that memoryless approaches such as the one proposed here can achieve performances on par with the memoryreplay-based approaches used as benchmarks, without the additional memory overhead.

Original: This suggests the need to identify more challenging continual learning assays.
Revised: This suggests the need to identify more challenging continual learning assays.

Original: Our approach allowed us to explore the transferability of optimal learning conditions across datasets and tasks, in order to understand the interplay between task-heterogeneity and continual learning across very different tasks.
Revised: Our approach allowed us to explore the transferability of optimal learning conditions across datasets and tasks, in order to understand the interplay between task-heterogeneity and continual learning across very different tasks.

Original: We demonstrated through systematic experiments that the accuracy of this transfer metalearning to datasets previously not seen can be largely explained through a transfer coefﬁcient that can be based on metrics of dimensionality and distance between datasets.
Revised: We demonstrated through systematic experiments that the accuracy of this transfer metalearning to datasets previously not seen can be largely explained through a transfer coefﬁcient that can be based on metrics of dimensionality and distance between datasets.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported through the Lifelong Learning
Revised: This work was supported through the Lifelong Learning

Original: Machines (L2M) program from DARPA/MTO.
Revised: Machines (L2M) program from DARPA/MTO.

Original: The material is also based in part by work supported by the U.S.
Revised: The material is also based in part by work supported by the U.S.

Original: Department of Energy, Ofﬁce of Science, under contract
Revised: Department of Energy, Ofﬁce of Science, under contract

Original: DE-AC02-06CH11357.
Revised: DE-AC02-06CH11357.

Revised: We gratefully acknowledge the computing resources provided on Bebop (and/or Blues), a high-performance computing cluster operated by the Laboratory Computing Resource Center at Argonne National

Revised: Laboratory.