Original: S ELF -A CTIVATING N EURAL E NSEMBLES FOR C ONTINUAL R EINFORCEMENT L EARNING
Revised: S ELF -A CTIVATING N EURAL E NSEMBLES FOR C ONTINUAL R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability of an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents.
Revised: The ability of an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents.

Original: Most methods devised to address this problem depend heavily on well-deﬁned task boundaries which simplify the problem considerably.
Revised: Most methods devised to address this problem depend heavily on well-deﬁned task boundaries which simplify the problem considerably.

Original: Our task-agnostic method, Self-Activating Neural Ensembles (SANE), uses a hierarchical modular architecture designed to avoid catastrophic forgetting without making any such assumptions.
Revised: Our task-agnostic method, Self-Activating Neural Ensembles (SANE), uses a hierarchical modular architecture designed to avoid catastrophic forgetting without making any such assumptions.

Original: At each timestep a path through the SANE tree is activated; during training only activated nodes are updated, ensuring that unused nodes do not undergo catastrophic forgetting.
Revised: At each timestep a path through the SANE tree is activated; during training only activated nodes are updated, ensuring that unused nodes do not undergo catastrophic forgetting.

Original: Additionally, new nodes are created as needed, allowing the system to leverage and retain old skills while growing and learning new ones.
Revised: Additionally, new nodes are created as needed, allowing the system to leverage and retain old skills while growing and learning new ones.

Original: We demonstrate our approach on MNIST and a set of grid world environments, demonstrating that SANE does not undergo catastrophic forgetting where existing methods do.
Revised: We demonstrate our approach on MNIST and a set of grid world environments, demonstrating that SANE does not undergo catastrophic forgetting where existing methods do.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Lifelong learning is of critical importance for the ﬁeld of robotics; an agent that interacts with the world should be able to continuously learn from it, acting intelligently in a wide variety of situations.
Revised: Lifelong learning is of critical importance for the ﬁeld of robotics; an agent that interacts with the world should be able to continuously learn from it, acting intelligently in a wide variety of situations.

Original: In marked contrast to this ideal, most standard deep reinforcement learning methods are centered around a single task.
Revised: In marked contrast to this ideal, most standard deep reinforcement learning methods are centered around a single task.

Original: First, a task is deﬁned, then a policy is learned to maximize the rewards the agent receives in that setting.
Revised: First, a task is deﬁned, then a policy is learned to maximize the rewards the agent receives in that setting.

Original: If the task is changed, a completely new model is learned, throwing away the previous model and previous interactions.
Revised: If the task is changed, a completely new model is learned, throwing away the previous model and previous interactions.

Original: Task speciﬁcation therefore plays a central role in current end-to-end deep reinforcement learning frameworks.
Revised: Task speciﬁcation therefore plays a central role in current end-to-end deep reinforcement learning frameworks.

Original: But is task-driven learning scalable?
Revised: But is task-driven learning scalable?

Original: In contrast, humans do not require concrete task boundaries to be able to effectively learn separate tasks – instead, we perform continual (lifelong) learning.
Revised: In contrast, humans do not require concrete task boundaries to be able to effectively learn separate tasks – instead, we perform continual (lifelong) learning.

Original: The same model is used to learn new skills, leveraging the lessons of previous skills to learn more efﬁciently, without forgetting old behaviors.
Revised: The same model is used to learn new skills, leveraging the lessons of previous skills to learn more efﬁciently, without forgetting old behaviors.

Original: However, when placed into continual learning settings, current deep reinforcement learning approaches do neither: the transfer properties of these systems are negligible and they suffer from catastrophic forgetting (McCloskey & Cohen, 1989; French, 2006).
Revised: However, when placed into continual learning settings, current deep reinforcement learning approaches do neither: the transfer properties of these systems are negligible and they suffer from catastrophic forgetting (McCloskey & Cohen, 1989; French, 2006).

Original: The core issue of catastrophic forgetting is that a neural network trained on one task starts to forget what it knows when trained on a second task, and this issue only becomes exacerbated as more tasks are added.
Revised: The core issue of catastrophic forgetting is that a neural network trained on one task starts to forget what it knows when trained on a second task, and this issue only becomes exacerbated as more tasks are added.

Original: The problem ultimately stems from training one network end-to-end sequentially; the shared nature of the weights and the backpropagation used to update them mean that later tasks overwrite earlier ones (McCloskey & Cohen, 1989; Ratcliff, 1990).
Revised: The problem ultimately stems from training one network end-to-end sequentially; the shared nature of the weights and the backpropagation used to update them mean that later tasks overwrite earlier ones (McCloskey & Cohen, 1989; Ratcliff, 1990).

Original: To handle this, past approaches have attempted a wide variety of ideas: from task-based regularization (Kirkpatrick et al., 2017), to learning different sub-modules for different tasks (Rusu et al., 2016), to dual-system slow/fast learners inspired by the human hippocampus (Schwarz et al., 2018).
Revised: To handle this, past approaches have attempted a wide variety of ideas: from task-based regularization (Kirkpatrick et al., 2017), to learning different sub-modules for different tasks (Rusu et al., 2016), to dual-system slow/fast learners inspired by the human hippocampus (Schwarz et al., 2018).

Original: The core problem of continual learning, which none of these methods address, is that the agent needs to autonomously determine how and when to adapt to changing environments, as it is infeasible for a human to indeﬁnitely provide an agent with task-boundary supervision.
Revised: The core problem of continual learning, which none of these methods address, is that the agent needs to autonomously determine how and when to adapt to changing environments, as it is infeasible for a human to indeﬁnitely provide an agent with task-boundary supervision.

Original: Speciﬁcally, these approaches rely on the notion of tasks to identify when to spawn new sub-modules, when to freeze weights, when to save parameters, etc.
Revised: Speciﬁcally, these approaches rely on the notion of tasks to identify when to spawn new sub-modules, when to freeze weights, when to save parameters, etc.

Original: Leaning on task boundaries is unscalable and side-steps the core problem.
Revised: Leaning on task boundaries is unscalable and side-steps the core problem.

Revised: There are a few existing task-free methods.

Revised: Some address the problem by utilizing ﬁxed subdivisions of the input space (Aljundi et al., 2018; Veness et al., 2019), which we believe limits their ﬂexibility.

Original: While recent experience-based approaches such as Rolnick et al.
Revised: Recent experience-based approaches such as Rolnick et al.

Original: (2018) forgo explicit task IDs,  they still maintain explicit experience replay of all the tasks to avoid overﬁtting. However, large replay buffers are unscalable with an ever-increasing number of tasks. 
Revised: (2018) forgo explicit task IDs, but to do so they must maintain a large replay buffer, which is unscalable with an ever-increasing number of tasks. To our knowledge only one other continual learning method,

Revised: Neural Dirichlet Process

Revised: Mixture Models (Lee et al., 2020), adaptively creates new clusters, and we show in our experiments that SANE outperforms it.

Original: Instead, our method approaches this problem head-on by dynamically adapting to changing environments.
Revised: Our method approaches the problem head-on by dynamically adapting to changing environments.

Original: Our task-agnostic hierarchical method, Self-Activating Neural Ensembles (SANE), depicted in Figure 1, is the core of the method.
Revised: Our task-agnostic hierarchical method, Self-Activating Neural Ensembles (SANE), depicted in Figure 1, is the core of the method.

Original: Every node in the tree is a separate module, none task-speciﬁc.
Revised: Every node in the tree is a separate module, none task-speciﬁc.

Original: At each timestep a single path through the tree is activated to determine the action to take.
Revised: At each timestep a single path through the tree is activated to determine the action to take.

Original: Only activated nodes are updated, leaving unused modules unchanged and therefore immune to catastrophic forgetting.
Revised: Only activated nodes are updated, leaving unused modules unchanged and therefore immune to catastrophic forgetting.

Original: Critically, our tree is dynamic; new nodes are created when existing nodes are found to be insufﬁcient.
Revised: Critically, our tree is dynamic; new nodes are created when existing nodes are found to be insufﬁcient.

Original: This allows for the creation of new paths through the tree for novel scenarios, preventing destructive parameter updates to the other nodes.
Revised: This allows for the creation of new paths through the tree for novel scenarios, preventing destructive parameter updates to the other nodes.

Original: SANE provides the following desirable properties for continual reinforcement learning: (a) It mitigates catastrophic forgetting by only updating relevant modules; (b) Because of its task-agnostic nature, unlike previous approaches, it does not require explicit supervision with task IDs; (c) It achieves these targets with bounded resources and computation.
Revised: SANE provides the following desirable properties for continual reinforcement learning: (a) It mitigates catastrophic forgetting by only updating relevant modules; (b) Because of its task-agnostic nature, unlike previous approaches, it does not require explicit supervision with task IDs; (c) It achieves these targets with bounded resources and computation.

Original: In our results, we demonstrate the ability of SANE to learn and retain MNIST digits when presented fully sequentially, a challenging task our baselines struggle with.
Revised: In our results, we demonstrate the ability of SANE to learn and retain MNIST digits when presented fully sequentially, a challenging task our baselines struggle with.

Original: We also demonstrate SANE on a series of three grid world tasks, showing that SANE works in the reinforcement learning setting.
Revised: We also demonstrate SANE on a series of three grid world tasks, showing that SANE works in the reinforcement learning setting.

Original:  M ODEL
Revised:  M ODEL

Original: The structure of our Self-Activating Neural Ensembles (SANE) system is a tree, as shown in Figure 1.
Revised: The structure of our Self-Activating Neural Ensembles (SANE) system is a tree, as shown in Figure 1.

Original: Each node in the tree contains both an actor and a critic, described in 2.1.
Revised: Each node in the tree contains both an actor and a critic, described in 2.1.

Original: At each timestep we activate a path through the tree from the root to a leaf, at each layer selecting the node whose critic estimates the highest activation score (described in 2.2).
Revised: At each timestep we activate a path through the tree from the root to a leaf, at each layer selecting the node whose critic estimates the highest activation score (described in 2.2).

Original: We then sample an action from that leaf node’s policy to execute in the environment.
Revised: We then sample an action from that leaf node’s policy to execute in the environment.

Original: During training, the activated nodes are the only ones updated; all other nodes are unchanged.
Revised: During training, the activated nodes are the only ones updated; all other nodes are unchanged.

Original: This enables behaviors selected for by other paths through the tree to remain unchanged.
Revised: This enables behaviors selected for by other paths through the tree to remain unchanged.

Original: The tree structure is critical to the success of the system; a node’s children can be seen as partitioning the input space that activates the node, specializing their policies to these subspaces.
Revised: The tree structure is critical to the success of the system; a node’s children can be seen as partitioning the input space that activates the node, specializing their policies to these subspaces.

Original: When they differ sufﬁciently from their parent in performance, they are promoted to a higher level of the tree and obtain children of their own, further partitioning the space.
Revised: When they differ sufﬁciently from their parent in performance, they are promoted to a higher level of the tree and obtain children of their own, further partitioning the space.

Original: Nodes are constantly being created, promoted to higher levels, or merged as the situation demands (described in Section 2.4).
Revised: Nodes are constantly being created, promoted to higher levels, or merged as the situation demands (described in Section 2.4).

Original: Static vs. Dynamic Trees: One critical question to ask is why use a dynamic structure and not just a huge static tree with only parameter updates?
Revised: Static vs. Dynamic Trees: One critical question to ask is why use a dynamic structure and not just a huge static tree with only parameter updates?

Original: In the case of the static tree, if a path through the network is beneﬁcial for one task, it is likely to be chosen as a starting point for the subsequent task, and updated in-place for this new setting.
Revised: In the case of the static tree, if a path through the network is beneﬁcial for one task, it is likely to be chosen as a starting point for the subsequent task, and updated in-place for this new setting.

Original: An example of this is in Jacobs et al.
Revised: Examples of this can be found in Jacobs et al.

Original: (1991), where specialized losses are necessary to distribute activation across the ensemble of experts, even in the case of a stationary distribution.
Revised: ; Shazeer et al. (2017), where specialized losses are necessary to distribute activation across the ensemble of experts, even in the case of a stationary distribution.

Original: With SANE, the change of environment would be detected and a new path created, thus minimizing disruption of the previously beneﬁcial behaviors.
Revised: With SANE, the change of environment would be detected and a new path created, thus minimizing disruption of the previously beneﬁcial behaviors.

Original: 2. N ODE C ONTENTS
Revised: 2. N ODE C ONTENTS

Original: Every node i contains the following: a policy ppp i , a critic C i , a replay buffer containing a list of (state s , discounted reward r disc ), and a count of the number of times this node has been used (usage count).
Revised: Every node i contains the following: a policy ppp i , a critic C i , a replay buffer containing a list of (state s , discounted reward r disc ), and a count of the number of times this node has been used (usage count).

Original: The policy ppp i presented here is a vector of n logits, where n is the number of actions available to the agent.
Revised: The policy ppp i presented here is a vector of n logits, where n is the number of actions available to the agent.

Original: The critic C i is a neural net mapping a state s to two scalars: the value estimate of the discounted reward received if the current node is activated, v i ( s ) , and an estimate of the absolute error:
Revised: The critic C i is a neural net mapping a state s to two scalars: the value estimate of the discounted reward received if the current node is activated, v i ( s ) , and an estimate of the absolute error:

Original: [Equation]
Revised: [Equation]

Original: With these we can deﬁne an optimistic estimate (upper conﬁdence bound) and a pessimistic estimate (lower conﬁdence bound) for what reward a node can achieve:
Revised: With these we can deﬁne an optimistic estimate (upper conﬁdence bound) and a pessimistic estimate (lower conﬁdence bound) for what reward a node can achieve:

Original: [Equation]
Revised: [Equation]

Original: α is a hyper-parameter representing how wide a margin around the expected value we allow.
Revised: α is a hyper-parameter representing how wide a margin around the expected value we allow.

Original: v UCB is analogous to an upper conﬁdence bound in multi-armed bandit literature: as training proceeds and v i becomes more accurate, u i will decrease, providing a natural transition from exploration to exploitation for each node as it learns. 
Revised: v UCB is analogous to an upper conﬁdence bound in multi-armed bandit literature: as training proceeds and v i becomes more accurate, u i will decrease, providing a natural transition from exploration to exploitation for each node as it learns. This may however act too optimistically in highly stochastic environments, an area of future work.

Original: Each node’s replay buffer contains the history of experiences per node and is used to train the node’s critic.
Revised: Each node’s replay buffer contains the history of experiences per node and is used to train the node’s critic.

Original: The usage count is the number of times a node has been activated and is used in the merging of nodes.
Revised: The usage count is the number of times a node has been activated and is used in the merging of nodes.

Original: These are described more in sections 2.3 and 2.4.
Revised: These are described more in sections 2.3 and 2.4.

Original: 2. I NFERENCE
Revised: 2. I NFERENCE

Original: For each observed state s t from the environment, we activate a path through the tree from the root to a leaf node, whose policy is used to determine the next action to take in the environment, a t .
Revised: For each observed state s t from the environment, we activate a path through the tree from the root to a leaf node, whose policy is used to determine the next action to take in the environment, a t .

Original: The environment then responds with a reward r t and a new observation s t +

Revised: The environment then responds with a reward r t and a new observation s t +

Original: (See Figure 2).
Revised: (See Figure 2).

Original: From an activated node, the next node to activate is selected from its children by taking the node with the highest optimistic upper bound of reward.
Revised: We ﬁnd the path recursively: from an activated node, the next node to activate is selected from its children by taking the node with the highest optimistic upper bound of reward.

Original: This process continues until a leaf node is reached.
Revised: This process continues until a leaf node is reached.

Original: After the set of activated nodes is determined, an action is selected stochastically from the activated leaf’s policy, and executed in the environment.
Revised: After the set of activated nodes is determined, an action is selected stochastically from the activated leaf’s policy, and executed in the environment.

Original: New nodes are not created mid-episode to keep the tree idempotent, which allows us to readily run environments in parallel.
Revised: New nodes are not created mid-episode to keep the tree idempotent, which allows us to readily run environments in parallel.

Original: During inference, three things happen at each timestep to support updating the tree.
Revised: During inference, three things happen at each timestep to support updating the tree.

Original: All active nodes:
Revised: All active nodes:

Original: • are registered to receive the replay entry for that timestep after the episode completes.
Revised: • are registered to receive the replay entry for that timestep after the episode completes.

Original: • compare their critic’s estimate to their parent’s.
Revised: • compare their critic’s estimate to their parent’s.

Original: If they differ sufﬁciently (Section 2.4), the child will be promoted during the next tree structure update.
Revised: If they differ sufﬁciently (Section 2.4), the child will be promoted during the next tree structure update.

Original: • increment their usage count.
Revised: • increment their usage count.

Original: 2. P ARAMETER U PDATES
Revised: 2. P ARAMETER U PDATES

Original: SANE training updates consist of two steps: parameter updates (replay buffer, policy, and critic) and tree structure updates (creating new nodes, merging nodes, etc.).
Revised: SANE training updates consist of two steps: parameter updates (replay buffer, policy, and critic) and tree structure updates (creating new nodes, merging nodes, etc.).

Original: After T steps of data collection have occurred, we trigger both a parameter update and a tree structure update.
Revised: After T steps of data collection have occurred, we trigger both a parameter update and a tree structure update.

Original: Replay Buffer Update: Each node’s replay buffer contains the history of the times that node was activated, and is used to train the node’s critic.
Revised: Replay Buffer Update: Each node’s replay buffer contains the history of the times that node was activated, and is used to train the node’s critic.

Original: Each entry in the replay buffer contains an input state s and a discounted reward received r disc , which is also used in the training of the policies.
Revised: Each entry in the replay buffer contains an input state s and a discounted reward received r disc , which is also used in the training of the policies.

Original: The discounted reward is computed in the standard fashion, as in Espeholt et al.
Revised: The discounted reward is computed in the standard fashion, as in Espeholt et al.

Original: When the addition of new entries results in the replay buffer exceeding its max length, instead of removing the oldest entries, random entries are removed instead.
Revised: When the addition of new entries results in the replay buffer exceeding its max length, instead of removing the oldest entries, random entries are removed instead.

Original: This is to mitigate each critic from developing a recency bias.
Revised: This is to mitigate each critic from developing a recency bias.

Original: Policy Update: The policy for each node is trained on-line using REINFORCE (Williams, 1992) with Monte Carlo returns.
Revised: Policy Update: The policy for each node is trained on-line using REINFORCE (Williams, 1992) with Monte Carlo returns.

Original: Training the Critics: The critic for each node is trained off-line using the replay buffer.
Revised: Training the Critics: The critic for each node is trained off-line using the replay buffer.

Original: More speciﬁcally, we split a node’s replay buffer into batches each of size k .
Revised: More speciﬁcally, we split a node’s replay buffer into batches each of size k .

Original: The critic has two functions to minimize: v ( s ) (the expected value) and u ( s ) (the uncertainty as deﬁned in equation 1).
Revised: The critic has two functions to estimate: v ( s ) (the expected value) and u ( s ) (the uncertainty as deﬁned in equation 1).

Original: We use
Revised: We use

Original: L2 loss for both reward and error prediction.
Revised: L2 loss for both reward and error prediction.

Original: Note that this is the slowest step in the training of the
Revised: Note that this is the slowest step in the training of the

Original: SANE tree.
Revised: SANE tree.

Original: Since the nodes all use separate critics, this step can be effectively parallelized.
Revised: Since the nodes all use separate critics, this step can be effectively parallelized.

Original: We only train a critic when its node was activated at least once since the last time nodes were trained.
Revised: We only train a critic when its node was activated at least once since the last time nodes were trained.

Original: 2. S TRUCTURE U PDATES
Revised: 2. S TRUCTURE U PDATES

Original: One of the critical properties of SANE is that the tree structure updates dynamically.
Revised: One of the critical properties of SANE is that the tree structure updates dynamically.

Original: Promoting nodes enables existing behaviors to stay unchanged, and merging nodes allows us to keep memory and computation bounded, both depicted in Figure 3.
Revised: Promoting nodes enables existing behaviors to stay unchanged, and merging nodes allows us to keep memory and computation bounded, both depicted in Figure 3.

Original: The term promotion is used to refer to cloning a node and placing it in the tree one level up from the original.
Revised: The term promotion is used to refer to cloning a node and placing it in the tree one level up from the original.

Original: Merging is the process of combining nodes and is necessary to cap the computational and memory costs that scale with tree size.
Revised: Merging is the process of combining nodes and is necessary to cap the computational and memory costs that scale with tree size.

Original: Node Promotion: Drift is the term for when an environment is non-stationary, e.g. when the reward distribution or the state transition distribution are changing over time.
Revised: Node Promotion: Drift is the term for when an environment is non-stationary, e.g. when the reward distribution or the state transition distribution are changing over time.

Original: Often where drift occurs, catastrophic forgetting follows because networks update to the new setting, forgetting the old.
Revised: Often where drift occurs, catastrophic forgetting follows because networks update to the new setting, forgetting the old.

Original: To recognize drift, non-leaf nodes at the time of creation have their critic cloned and frozen, creating a static critic called the anchor .
Revised: To recognize drift, non-leaf nodes at the time of creation have their critic cloned and frozen, creating a static critic called the anchor .

Original: We compare the prediction of a node’s critic to the prediction its parent’s anchor makes.
Revised: We compare the prediction of a node’s critic to the prediction its parent’s anchor makes.

Original: We say that sufﬁcient change has occurred when the reward bounds predicted by a node’s critic do not include the value predicted by its parent’s anchor, which functions as a static baseline.
Revised: We say that sufﬁcient change has occurred when the reward bounds predicted by a node’s critic do not include the value predicted by its parent’s anchor, which functions as a static baseline.

Original: In such a scenario, the node should be promoted up a level of the tree, because we wish to capture the new situation being observed, and minimize disruption of previously learned behavior (catastrophic forgetting).
Revised: In such a scenario, the node should be promoted up a level of the tree, because we wish to capture the new situation being observed, and minimize disruption of previously learned behavior.

Original: Concretely, we say that sufﬁcient change has occurred when either of the following inequalities hold, for a given input state s , node n , and parent’s anchor a : v UCB,n ( s ) < v a ( s ) , v LCB,n ( s ) > v a ( s ) .
Revised: Concretely, we say that sufﬁcient change has occurred when either of the following inequalities hold, for a given input state s , node n , and parent’s anchor a : v UCB,n ( s ) < v a ( s ) , v LCB,n ( s ) > v a ( s ) .

Original: In order to avoid an extraneous number of promotions for outliers, a leaf node must have been used more than k m times since it was created or last promoted.
Revised: In order to avoid an extraneous number of promotions for outliers, a leaf node must have been used more than k m times since it was created or last promoted.

Revised: An implicit assumption here is that the change in reward received is sufﬁcient for distinguishing relevant changes in setting.

Revised: We believe this to be a general assumption; if, as the task changes, a node’s behavior results in the same received reward (and exploration does not expose a better path), then the change was not relevant for determining the optimal policy, and can be effectively ignored.

Original: Merging Nodes: To prevent unlimited memory consumption and increasingly long activation times, we limit the number of children each node can have.
Revised: Merging Nodes: To prevent unlimited memory consumption and increasingly long activation times, we limit the number of children each node can have.

Original: The limit is deﬁned per level of the tree: level k has limit (cid:96) k .
Revised: The limit is deﬁned per level of the tree: level k has limit (cid:96) k .

Original: We start by ﬁnding the two entries in the set that are closest in policy space.
Revised: We start by ﬁnding the two entries in the set that are closest in policy space.

Revised: Conceptually, this is so we can ﬁnd general patterns in the state space that elicit the same optimal policy.

Original: To do this we convert the policies from logits to probabilities, then ﬁnd the two policies that have the smallest 
Revised: To do this we convert the policies from logits to probabilities, then ﬁnd the two policies that have the smallest L2 distance.

Original: L2 distance.
Revised: L2 was selected because it is symmetric and can be computed batch-wise rapidly.

Original: The usage counts, policies, and replay buffers are combined by doing a weighted average of the nodes’ contents by usage count.
Revised: The usage counts, policies, and replay buffers are combined by doing a weighted average of the nodes’ contents by usage count, explained in more detail in section A.2.

Original: The critic for the new node is trained using the merged replay buffer, after being initialized from the critic of the selected node with higher usage.
Revised: The critic for the new node is trained using the merged replay buffer, after being initialized from the critic of the selected node with higher usage.

Original: Non-leaf nodes additionally update the anchor by combining the nodes’ anchors’ replay buffers and retraining once.
Revised: Non-leaf nodes additionally update the anchor by combining the nodes’ anchors’ replay buffers and retraining once.

Original: Merging nodes is the exception to the general rule of never updating the anchors.
Revised: Merging nodes is the exception to the general rule of never updating the anchors.

Original: Initialization: The ﬁrst node in the tree is initialized as a child of the root and has a random critic and a uniform policy.
Revised: Initialization: The ﬁrst node in the tree is initialized as a child of the root and has a random critic and a uniform policy.

Original: That node is cloned twice, which become the ﬁrst node’s children.
Revised: That node is cloned twice, which become the ﬁrst node’s children.

Original: This continues until the tree has reached the pre-speciﬁed maximum height.
Revised: This continues until the tree has reached the pre-speciﬁed maximum height.

Original: From now on, the height of the tree remains ﬁxed.
Revised: From now on, the height of the tree remains ﬁxed.

Original: To maintain this, whenever a node is promoted, that node is recursively cloned to generate children until the maximum height is re-achieved.
Revised: To maintain this, whenever a node is promoted, that node is recursively cloned to generate children until the maximum height is re-achieved.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present two types of experiments that demonstrate the behavior of SANE.
Revised: We present two types of experiments that demonstrate the behavior of SANE.

Original: The ﬁrst trains on the digits of MNIST sequentially, to show recall under complete domain shift.
Revised: The ﬁrst trains on the digits of MNIST sequentially, to show recall under complete domain shift.

Original: The second learns a series of grid world environments, to show that the method works in a reinforcement learning setting.
Revised: The second learns a series of grid world environments, to show that the method works in a reinforcement learning setting.

Revised: Both never re-train on a task they’ve seen before; this is to ensure there are concrete points of domain shift.

Original:  In these experiments we only consider two-layer SANE trees, with their sizes ([ (cid:96) 0 , (cid:96) 1 ]) indicating respectively the maximum number of nodes at the ﬁrst layer, and the maximum number of children each of those may have.
Revised: The environments are simple enough that this is not ensured by randomness alone. In these experiments we only consider two-layer SANE trees, with their sizes ([ (cid:96) 0 , (cid:96) 1 ]) indicating respectively the maximum number of nodes at the ﬁrst layer, and the maximum number of children each of those may have.

Original: A [20, 4] tree therefore has 20 top layer nodes and 80 second layer nodes, for a total of 100 nodes.
Revised: A [20, 4] tree therefore has 20 top layer nodes and 80 second layer nodes, for a total of 100 nodes.

Original: The experiments are all run over 4 seeds, and the error bars represent the standard error of the mean.
Revised: The experiments are all run over 4 seeds unless otherwise noted, and the error bars represent the standard error of the mean.

Original:  In both cases we use IMPALA (Espeholt et al., 2018) as the non-continual learning baseline, to demonstrate the effects of catastrophic forgetting, and CLEAR (Rolnick et al., 2018) as the continual learning baseline.
Revised: Baselines: In both cases we use IMPALA (Espeholt et al., 2018) as the non-continual learning baseline, to demonstrate the effects of catastrophic forgetting, and CLEAR (Rolnick et al., 2018) as a continual learning baseline.

Original: CLEAR is a variant of IMPALA that adds an extensive replay buffer to maintain experiences from old tasks.
Revised: CLEAR is a variant of IMPALA that adds an extensive replay buffer to maintain experiences from old tasks.

Original: In particular, we selected CLEAR because it is one of the few other methods that does not require task IDs as input.
Revised: In particular, we selected CLEAR because it is one of the few other methods that does not require task IDs as input.

Original: We chose for CLEAR the same number of entries in its replay buffer as the upper bound of the number SANE uses, across all nodes: 409,600 frames.
Revised: We chose for CLEAR the same number of entries in its replay buffer as the upper bound of the number SANE uses, across all nodes: 409,600 frames.

Original: All network and implementation details are given in the Appendix.
Revised: All network and implementation details are given in the Appendix.

Original: As SANE uses one network per critic, whereas IMPALA and CLEAR  each use only one network total, we can see thatthere are two versions of each of our baselines: one where they all use the same base network architecture, and one where IMPALA and CLEAR both use signiﬁcantly larger networks, such that the total number of parameters is approximately equal to the number across all nodes of  the SANE tree.
Revised: For the MNIST experiment we additionally use a Neural Dirichlet Process Mixture Model baseline (Lee et al., 2020), described further below. CLEAR has an implicit trade-off between how easily it learns a new task, and how much it recalls old ones, as deﬁned by the ratio of entries in each training batch that are selected from the replay buffer, versus newly acquired entries. Additionally, if we naively run CLEAR for the same number of timesteps as SANE, we might end up overﬁtting. We thus additionally consider two types of modiﬁcation to CLEAR: changing the replay ratio and early stopping.

Original: For MNIST we increase the networksize of IMPALA and CLEAR by a factor of 75 (indicated as ”75x” in the graphs), and for MiniGrid by a factor of 40 (”40x” in the graphs), forapproximate parity with the number ofnodes at steady state in the [20, 4] and [10, 4] trees respectively.
Revised: In the latter case we continue to evaluate the network for the rest of the allotted time, to maintain alignment.

Original: Environments/Datasets: We conduct two experiments: MNIST and MiniGrid.
Revised: Environments/Datasets: We conduct two experiments: MNIST and MiniGrid.

Original: MNIST allows us to test network behavior in the presence of huge domain shifts as well as a relatively large number of tasks, to highlight how approaches like CLEAR struggle in these scenarios.
Revised: MNIST allows us to test network behavior in the presence of huge domain shifts as well as a relatively large number of tasks, to highlight how approaches like CLEAR struggle in these scenarios.

Original: MiniGrid allows us to demonstrate how SANE avoids forgetting in the reinforcement learning setting.
Revised: MiniGrid allows us to demonstrate how SANE avoids forgetting in the reinforcement learning setting.

Original: Note that while previous efforts have used Atari, we do not use Atari for a couple of reasons: (a) this setting is particularly ill-suited for continual learning per the desiderata given in (Farquhar & Gal, 2018), for instance as there is little cross-task resemblance, task IDs can be readily inferred (b) the required compute is high (a bottleneck in new model development).
Revised: Note that while previous efforts have used Atari, we do not use Atari for a couple of reasons: (a) this setting is particularly ill-suited for continual learning per the desiderata given in (Farquhar & Gal, 2018), for instance as there is little cross-task resemblance, task IDs can be readily inferred (b) the required compute is high (a bottleneck in new model development).

Original: 3. MNIST E XPERIMENTS
Revised: 3. MNIST E XPERIMENTS

Original: MNIST (LeCun et al., 2010) is a dataset of the digits 0-9, each represented by a grayscale 28x image.
Revised: MNIST (LeCun et al., 2010) is a dataset of the digits 0-9, each represented by a grayscale 28x image.

Original: Demonstrating continual learning with MNIST has an extensive history; standard baselines such as permuted MNIST (Kirkpatrick et al., 2017; Schwarz et al., 2018; Goodfellow et al., 2015) train on the full set of digits concurrently, adding signiﬁcant pixel-level shifts to change the appearance between tasks.
Revised: Demonstrating continual learning with MNIST has an extensive history. Permuted MNIST (Kirkpatrick et al., 2017; Schwarz et al., 2018; Goodfellow et al., 2015) trains on the full set of digits concurrently, adding signiﬁcant pixel-level shifts to change the appearance between tasks.

Original: However, while the domain may shift dramatically, the distribution over actions does not.
Revised: Split MNIST (Farquhar & Gal, 2018) splits the 10 digits into 5 sequential tasks of 2 digits each. Both have assumptions we can now relax; in the former case, the assumption of balanced classes, and in the latter the existence of negative examples for each class being trained.

Original: We  introduce sequential MNIST, an alternative task in which digits are trained completely sequentially: ﬁrst the 0s, then the 1s, etc.
Revised: We therefore introduce Sequential MNIST, an alternative task in which digits are trained completely sequentially: ﬁrst the 0s, then the 1s, etc.

Original: Aside from being a sequence oftasks with a fundamentally different distribution over desirable outputs, we believe that this setting is signiﬁcantly more
Revised: To convert this classiﬁcation task to an RL task, we deﬁne the action to be the selection of a class (i.e. the action space is of size 10), and the reward is deﬁned to be 1 if the correct class is selected,otherwise.

Original: Method 00- 00- 00- 0SANE 97.
Revised: We train for 300,000 frames per digit.

Original: [Table] [Table]
Revised: No normalization is done other than scaling pixel intensities to between [0, 1].

Original: IMPALA
Revised: For this experiment we additionally run a Neural Dirichlet Process Mixture Model (Lee et al., 2020) as a baseline, using the code provided by the authors.

Original: [Table] [Table] [Table]
Revised: This is a classiﬁcation-only method, so for this baseline we do not do the conversion to an RL task described above.

Original: IMPALA 100x 100.
Revised: We use the default MNIST conﬁguration provided in their code. Due to time constraints NDPM is shown with only one run, though preliminary experiments indicated consistent results.

Original:  CLEAR 100. CLEAR 100x [Table] [Table]
Revised: Additionally, we experimented with different conﬁgurations of CLEAR by tuning the replay ratio parameter and by early stopping (see

Original: Mem Last [Table]
Revised: Appendix A.7), and selected a replay fraction of 0.875 for our CLEAR baseline.

Original: The results are presented in Figure 4.
Revised: The results are presented in Figure 4.

Original: The networks are trained on the given digit during the timesteps indicated by the shaded region.
Revised: The networks are trained on the given digit during the timesteps indicated by the shaded region.

Original: In the non-shaded regions of the plots, the networks are evaluated on that digit alone, to analyze recall.
Revised: In the non-shaded regions of the plots, the networks are evaluated on that digit alone, to analyze recall.

Original: As this ﬁgure visualizes recall and not generalization, these evaluation steps are done on the MNIST Train dataset.
Revised: As this ﬁgure visualizes recall and not generalization, these evaluation steps are done on the MNIST Train dataset.

Original: Additionally we test the methods by evaluating them on the Test dataset for the digits cumulative to the most recent digit seen during training.
Revised: Additionally we test the methods by evaluating them on the Test dataset for the digits cumulative to the most recent digit seen during training.

Original: These results are presented in Table 1. ”
Revised: These results are presented in Table 1. ”

Original: Mem Last” is  provided asa baseline forwhat the expected results would be if the network always identiﬁes the last-seen digit.
Revised: Mem Last” is [Table] provided as for

Original: Onlythe SANEmethod consistently recalls digits after their training period, not experiencing catastrophic forgetting.
Revised: Only SANE consistently recalls digits after their training period, avoiding experiencing catastrophic forgetting.

Original: This setting is particularly challenging for typical methods, because in each training regime there is always only one correct action to take, and it is easy to overﬁt to providing the singular desired answer.
Revised: This setting is particularly challenging for typical methods, because in each training regime there is always only one correct action to take, and it is easy to overﬁt to providing the singular desired answer.

Original: It is worth noting that SANE did not perform ﬂawlessly; there are a few digits (5, 8, 9) that SANE learns inconsistently in the allotted number of frames.
Revised: It is worth noting that SANE did not perform ﬂawlessly; there are a few digits (5, 8, 9) that SANE learns inconsistently in the allotted number of frames.

Original: There are a few speciﬁc behaviors to note. (a) CLEAR’s added replay buffer does provide beneﬁt for recalling ”0” during the training of future digits, though the beneﬁtquickly tapers off.
Revised: There are a few speciﬁc behaviors to note: (a) CLEAR’s added replay buffer does provide beneﬁt for recall of earlier digits, though the beneﬁt tapers off as more digits are observed.

Original: (b) SANE’s curves are reliably cyclic: when SANE begins to do well on a  digit, its performance on prior digits decreases.
Revised: (b) SANE’s curves are reliably cyclic: when SANE begins to do well on a new digit, its performance on prior digits decreases.

Original: However, when the next new digit is presented, the performance on prior digits increases again.
Revised: However, when the next new digit is presented, the performance on prior digits increases again.

Original: The network is  overﬁtting while SANE masters a digit, then the negative examples from the start of training  the next digit compensate for the previous overﬁtting, giving us some reverse transfer.
Revised: Our hypothesis is that this is due to overﬁtting on the current digit, which is mitigated by the negative examples when training on the next digit, leading to some reverse transfer.

Original: (c) In the cumulative test experiment, SANE not only consistently produces the highest scores, but the results are not monotonically decreasing.
Revised: (c) SANE’s performance in the Test condition plateaus lower than the other methods; this is due to the application of a tanh to the logits of the policy to improve stability, meaning even in Test every action has a chance of selection, described further in section A.2.

Original: (d) Increasing the size of the baseline networks did not improve performance.
Revised: (d) NDPM learns each digit less well than the previous, with only digit 0 actually being mastered, and even that digit still experiences more forgetting than SANE.

Original: 3. M INI G RID E XPERIMENTS
Revised: 3. M INI G RID E XPERIMENTS

Original: MiniGrid (Chevalier-Boisvert et al., 2018) is a lightweight grid world library designed to procedurally generate environments.
Revised: MiniGrid (Chevalier-Boisvert et al., 2018) is a lightweight grid world library designed to procedurally generate environments.

Original: We train for 500,000 frames on the Empty environment, followed by 750,000 frames on the 2-Room environment, and ﬁnally 1 million frames on the Unlock environment,all depicted in Figure 5.
Revised: We train for 750,000 frames on the 2 Room environment, followed by 1.5 million frames on the Unlock environment, and ﬁnally 1.5 million frames on the Key-Corridor environment, depicted in Figure 5.

Original: The Empty environment isalways initialized the same way, whereas 2-Room and Unlock are randomized at the completion of each episode.
Revised: Each is randomized at the completion of each episode.

Original: The observation is an egocentric representation of the environment, capturing state information about the 7x7 square in front of the agent.
Revised: The observation is an egocentric representation of the environment, capturing state information about the 7x7 square in front of the agent.

Original: The training results are presented in Figure 5.
Revised: The training results are presented in Figure 5.

Original: As with MNIST, the shaded region indicates when that environment was being trained, while the unshaded parts are evaluation steps on that environment, to visualize recall.
Revised: As with MNIST, the shaded region indicates when that environment was being trained, while the unshaded parts are evaluation steps on that environment, to visualize recall.

Original: The Empty8x8 graph is the most insightful: CLEAR has almost perfect recall of this environment while 2-Room is training, but catastrophically forgets it during the training of Unlock. In contrast, both SANE trees do see a decrease in  performancewhile 2-Room is training, but the larger [15, 4] tree in particular does not exhibit catastrophic forgetting, exhibited by its continued positive reward during recall.
Revised: We experimented with different conﬁgurations of CLEAR (see Appendix A.7), and for MiniGrid selected early stopping, though a replay ratio of 0.75 performed comparably. SANE’s performance is comparable to that of CLEAR with early stopping, indicating that SANE can successfully mitigate catastrophic forgetting in RL environments, even improving on performance in places. CLEAR without early stopping performs worse than both, indicating that it is likely overﬁtting.

Original: We can also see the beneﬁt thatlarger SANEtrees can provide; it is likely the smaller one ran out of capacity. It suggests that as we continue to scale SANE up, we may see increases in performance.
Revised: The fact that SANE trains extensively on the Unlock environment after plateauing (for approximately an additional 1 million timesteps), may indicate that SANE is less susceptible to overﬁtting, potentially another beneﬁt of the modularity of our architecture.

Revised: This beneﬁt is not to be understated; without inherent resilience to overﬁtting, agents that are continuously acting in the world must have some automatic mechanism to detect it, a feature which most CL methods do not presently have, with the exception of (Aljundi et al., 2018).

Original: The 2-Room training curve simply shows that both SANE methods and both CLEAR methods are performing about the same in this setting, with IMPALA catastrophically forgetting as expected.
Revised: IMPALA exhibits catastrophic forgetting as expected, even though it struggles with learning the second two environments.

Original: The Unlock graph shows that the larger SANE tree learns more sample efﬁciently than its smaller sibling, though not as rapidly as CLEAR.
Revised: Creation of new nodes: Figure 6 shows when SANE decided to promote nodes during one run of each experiment. For MiniGrid, it steadily increases per task, whereas for MNIST the pattern is less distinct. The former behavior is expected; low as negative examples are provided to the current best nodes, then high as nodes improve over their parents.

Original: Interestingly, CLEAR learns readily while IMPALA does not. Again, as with MNIST, we see that increasing the size of the baseline networks provides no beneﬁt.
Revised: It may remain high because the tree starts to run out of capacity around 300k steps, and promotion+merging nodes is less sample efﬁcient than promotion alone. The behavior of MNIST is less clear though, and future work will investigate it more closely.

Original: Finally, we provide some  intuition for the behavior of the SANE network by visualizing a few states that activate the same node in Figure 6, located in the Appendix. 
Revised: We provide some further intuition for the behavior of the SANE network by visualizing a few states that activate the same node in Figure 7, located in the Appendix. Parameter and architecture analysis: Unfortunately SANE is relatively slow (about a day to train

Revised: Minigrid, versus an hour for CLEAR) and considerably compute-intensive, so a complete grid search and set of ablations has been out of reach.

Revised: Informal experimentation has shown little sensitivity to most parameters, with the primary exceptions being the size of the nodes and the number of epochs they are trained for, as well as (cid:15) in (cid:15) -greedy.

Revised: The former two are particularly inconvenient because they are highly tied to the runtime of the method.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORK

Original: The goal of continual learning, or lifelong learning, is to create an agent that can learn to accomplish a wide variety of tasks sequentially (Lesort et al., 2019).
Revised: The goal of continual learning, or lifelong learning, is to create an agent that can learn to accomplish a wide variety of tasks sequentially (Lesort et al., 2019).

Original: To accomplish this a system must balance stability (the extent to which existing knowledge is retained) and plasticity (how readily new knowledge is acquired) (Parisi et al., 2018).
Revised: To accomplish this a system must balance stability (the extent to which existing knowledge is retained) and plasticity (how readily new knowledge is acquired) (Parisi et al., 2018).

Original: The former in particular has posed a substantial problem for neural networks, which undergo catastrophic forgetting, tending to completely forget old tasks when presented with new ones, often quite abruptly (French, 2006).
Revised: The former in particular has posed a substantial problem for neural networks, which undergo catastrophic forgetting, tending to completely forget old tasks when presented with new ones, often quite abruptly (French, 2006).

Original: Attempts to correct for this tend to fall into these broad categories, which we discuss below: dynamic networks, dual memory, regularization, rehearsal, sparse-coding, and ensembles (Parisi et al., 2018; Kemker et al., 2017).
Revised: Attempts to correct for this tend to fall into these broad categories, which we discuss below: dynamic networks, dual memory, regularization, rehearsal, sparse-coding, and ensembles (Parisi et al., 2018; Kemker et al., 2017).

Original: Each direction has received a signiﬁcant amount of work, and our overview here will be non-exhaustive.

Original: In particular we largely focus here on methods that have been demonstrated on reinforcement learning environments.

Original: Dynamic architectures and dual memory: One example of dynamic architectures, where resources are added as-needed, is CHILD (Ring, 1997), which determines when to create new units by analyzingthe average errors observed in the weights of the network.
Revised: Dynamic architectures and dual memory: One example of dynamic architectures, where resources are added as-needed, is CHILD (Ring, 1997), which determines when to create new units by analyzing average errors observed in network weights.

Original: Another dynamic architecture method is Progressive Networks (Rusu et al., 2016), where new neural networks (”columns”) are added as new tasks are encountered, with lateral connections created to the networks of previously learned tasks.
Revised: Another dynamic architecture method is Progressive Networks (Rusu et al., 2016), where new neural networks (”columns”) are added as new tasks are encountered, with lateral connections created to the networks of previously learned tasks.

Original: Progress and Compress (Schwarz et al., 2018) adapts this into a dual-memory method byonly maintaining two columns and regularly distilling the active column into the knowledge-base column.
Revised: Progress and Compress (Schwarz et al., 2018) adapts this into a dual-memory method by maintaining two columns and regularly distilling the active column into the knowledge-base column.

Original: Regularization: Elastic Weight Consolidation (Kirkpatrick et al., 2017) and Synaptic Intelligence (Zenke et al., 2017) are examples of regularization.
Revised: Regularization: Elastic Weight Consolidation (Kirkpatrick et al., 2017) and Synaptic Intelligence (Zenke et al., 2017) are examples of regularization.

Original: Both automatically learn an importance measure for the individual neuron weights for particular tasks, such that new tasks minimally inﬂuence weights that were important for prior tasks.
Revised: Both automatically learn an importance measure for the individual neuron weights for particular tasks, such that new tasks minimally inﬂuence weights that were important for prior tasks.

Original:  Rehearsal: One rehearsal method is CLEAR (Rolnick et al., 2018), which maintains a replay buffer over old tasks, using reservoir sampling and a behavioral cloning loss to ensure the network continues to respond appropriately to the inputs of old tasks, giving this method some regularization as well.
Revised: The method presented by Aljundi et al. (2018) is a taskfree regularization method that uses features of the loss as it is updated: when a plateau is detected after a peak, the parameter importance weights are updated. Rehearsal: One rehearsal method is CLEAR (Rolnick et al., 2018), which maintains a replay buffer over old tasks, using reservoir sampling and a behavioral cloning loss to ensure the network continues to respond appropriately to the inputs of old tasks, giving this method some regularization as well.

Original: CLEAR is built on top of IMPALA (Espeholt et al., 2018), an asynchronous non-continual learning actor-critic method.
Revised: CLEAR is built on top of IMPALA (Espeholt et al., 2018), an asynchronous non-continual learning actor-critic method.

Revised: Another rehearsal method is Maximally Interfered Retrieval (Aljundi et al., 2019), which trains on the entries from its buffer that a proposed update (using recently observed data) most negatively impact.

Original:  Sparse coding and ensembles: Fixed Expansion Layer Networks (Coop et al., 2013) (FEL) is an example of both a sparse-coding and an ensemble method, though not one demonstrated in RL environments; it introduces a layer that is sparsely activated, such that the error signal does not propagate through all weights, and creates an ensemble of these to improve results.
Revised: MIR also introduces a variant using a generative model. Both of these do not require task IDs. Sparse coding and ensembles: Fixed Expansion Layer Networks (Coop et al., 2013) (FEL) is an example of both a sparse-coding and an ensemble method, though not one demonstrated in RL environments; it introduces a layer that is sparsely activated, such that the error signal does not propagate through all weights, and creates an ensemble of these to improve results.

Original: PathNet (Fernando et al., 2017), which may also be considered an ensemble method (Kemker et al., 2017), uses an evolutionary strategy to select a path through the network.
Revised: PathNet (Fernando et al., 2017), which may also be considered an ensemble method (Kemker et al., 2017), uses an evolutionary strategy to select a path through the network.

Original: The usage of speciﬁc paths instead of the whole network is a similarity PathNet shares with SANE.
Revised: The usage of speciﬁc paths instead of the whole network is a similarity PathNet shares with SANE.

Original: However, the method still requires task boundaries to determine when to freeze paths, a key difference from SANE. 
Revised: However, the method still requires task boundaries to determine when to freeze paths, a key difference from SANE. Task-boundary free:

Revised: In addition to CLEAR, MIR, and Task-Free Continual Learning, there are a few other explicitly task-boundary free works.

Revised: Goyal et al.

Revised: (2019) also present a distributed network of primitives that independently determine the score used to activate them.

Revised: However, their primitives are constrained in terms of what part of the input space they see, and new primitives are not created over time.

Original: C ONCLUSION
Revised: Gated Linear Networks (Veness et al., 2019) introduces a hierarchical backpropagation-free method that gates neuron selection by associating inputs with predeﬁned context vectors. Continual Neural Dirichlet Process Mixture Models Lee et al. (2020) leverages the ability of standard DPMMs to handle clustering with an arbitrary number of clusters, increasing as necessary when new data is not well-explained by an existing cluster. Finally, (Zeno et al., 2019) approach task-free continual classiﬁcation by estimating a Bayesian posterior online.

Revised: Additionally, some of the components of SANE have occurred in non-CL literature.

Revised: For instance,

Revised: Mnih & Hinton (2009) and Shen et al.

Revised: (2017) utilize a method that selects a leaf node by making choices at each internal node of a tree, though tree creation is by clustering not by drift detection.

Revised: Ghosh et al.

Revised: (2017) presents Divide and Conquer, an idea conceptually similar to SANE; it partitions the input space and learns an ensemble of specialized policies, though they cluster with K-Means while SANE forms partitions automatically.

Revised:  C ONCLUSION

Original: The core challenge behind continual learning is knowing how and when to adapt to novel situations without overwriting previous behavior.
Revised: The core challenge behind continual learning is knowing how and when to adapt to novel situations without overwriting previous behavior.

Original: Most existing methods side-step the problem by allowing the task boundaries to be speciﬁed, giving a clear signal for when to add parameters, freeze them, reinitialize them, etc.
Revised: Most existing methods side-step the problem by allowing the task boundaries to be speciﬁed, giving a clear signal for when to add parameters, freeze them, reinitialize them, etc.

Original: Our method, Self-Activating Neural Ensembles, needs no such crutch.
Revised: Our method, Self-Activating Neural Ensembles, needs no such crutch.

Original: It automatically detects and adapts to drift in the environment by adding new nodes to the SANE tree.
Revised: It automatically detects and adapts to drift in the environment by adding new nodes to the SANE tree.

Original: Additionally, since only the necessary set of nodes is activated at each timestep, nodes that did not participate are not updated with the new interaction data and therefore are not susceptible to catastrophic forgetting.
Revised: Additionally, since only the necessary set of nodes is activated at each timestep, nodes that did not participate are not updated with the new interaction data and therefore are not susceptible to catastrophic forgetting.

Original: Finally, our method automatically merges nodes as necessary, ensuring the resources consumed remain bounded.
Revised: Finally, our method automatically merges nodes as necessary, ensuring the resources consumed remain bounded.

Original: The biggest roadblock to deploying SANE on more and harder problems is that presently during every update, all nodes’ critics are trained. While this step is parallelizable, it still forms the primary bottleneck and limits the size of the networks we can train.
Revised: Future Work:

Original: Improving this is oneclear direction for future work.
Revised: Improving the speed of training has been a bottleneck, and is one direction for future work.

Original: Other directions include experimenting withtrees with more layers, sharing parameters between nodes, having a common pre-encoder akin to the visual cortex, actively mining negative examples from other nodes,  and many others.
Revised: Other directions include experimenting with more layers, sharing parameters between nodes, having a common pre-encoder akin to the visual cortex, actively mining negative examples from other nodes, adapting to continuous and stochastic domains, leveraging distributional models, and many others.

Original: A. SANE I MPLEMENTATION D ETAILS
Revised: A. SANE I MPLEMENTATION D ETAILS

Original: There are a number of details left out of the main SANE explanation, to streamline the explanation down to the most signiﬁcant concepts.
Revised: There are a number of details left out of the main SANE explanation, to streamline the explanation down to the most signiﬁcant concepts.

Original: These details are explained here, roughly in decreasing order of signiﬁcance.
Revised: These details are explained here, roughly in decreasing order of signiﬁcance.

Original: • The entries in the replay buffer are importance-weighted when being used to train the critic.
Revised: • The entries in the replay buffer are importance-weighted when being used to train the critic.

Original: In particular they are scaled by the simple ratio: P cur ( a ) P old ( a ) , or in other words the ratio of the probability of the selected action using the node’s current policy to that if the node’s policy at the time the action was originally taken.
Revised: In particular they are scaled by the simple ratio: P cur ( a ) P old ( a ) , or in other words the ratio of the probability of the selected action using the node’s current policy to that if the node’s policy at the time the action was originally taken.

Original: • When merging nodes, only the ﬁrst 3/4 of the set of nodes is eligible to be merged.
Revised: • When merging nodes, only the ﬁrst 3/4 of the set of nodes is eligible to be merged.

Original: This is under the premise that the most recent nodes were likely newly created, and we’re giving them a little longer to be used before we collapse it into another node.
Revised: This is under the premise that the most recent nodes were likely newly created, and we’re giving them a little longer to be used before we collapse it into another node.

Original: • During training SANE operates in an epsilon-greedy manner, selecting a random action some (cid:15) fraction of the time, to encourage exploration.
Revised: • During training SANE operates in an epsilon-greedy manner, selecting a random action some (cid:15) fraction of the time, to encourage exploration.

Original: • All nodes’ policies are scaled as 3 tanh ( P ) , meaning each logit falls in the range (-3, 3).
Revised: • All nodes’ policies are scaled as 3 tanh ( P ) , meaning each logit falls in the range (-3, 3).

Original: This is for a few reasons.
Revised: This is for a few reasons.

Original: (1)  Even if a node is exceptionally conﬁdent in an action, when it encounters a new situation it must still have some non-zero chance of trying another action (related to epsilon-greedy).
Revised: (1) This stabilizes the REINFORCE gradient applied to the policy. Even if a node is exceptionally conﬁdent in an action, when it encounters a new situation it must still have some non-zero chance of trying another action (related to epsilon-greedy).

Original: (2) Raising the probability of a previously-disadvantaged action to a range where it gets selected frequently should not take a large number of examples.
Revised: (3) Raising the probability of a previously-disadvantaged action to a range where it gets selected frequently should not take a large number of examples.

Original: • We maintain 3 sets of usage counts: (1) The total number of times the node has been activated across its entire ancestry (i.e. including the count of the node it was cloned from), roughly representing how many unique replay entries the critic has been exposed to.
Revised: • We maintain 3 sets of usage counts: (1) The total number of times the node has been activated across its entire ancestry (i.e. including the count of the node it was cloned from), roughly representing how many unique replay entries the critic has been exposed to.

Original: During merge this count becomes the average of the two nodes’.
Revised: During merge this count becomes the average of the two nodes’.

Original: This is the usage used in weighting policies and replay buffers during merge.
Revised: This is the usage used in weighting policies and replay buffers during merge.

Original: The usage count since this node was created or last cloned.
Revised: The usage count since this node was created or last cloned.

Original: Used for determining whether the node is eligible to be cloned again.
Revised: Used for determining whether the node is eligible to be cloned again.

Original: The usage count since the last time the node was trained, only used to determine whether to train the node’s critic during the update step.
Revised: The usage count since the last time the node was trained, only used to determine whether to train the node’s critic during the update step.

Original: • Usage count (1) from the above point is not used directly in determining a node’s weight; instead the weight is given by max (1 , tanh ( usage count/ 20000) ∗ 100) .
Revised: • Usage count (1) from the above point is not used directly in determining a node’s weight; instead the weight is given by max (1 , tanh ( usage count/ 20000) ∗ 100) .

Original: These constants have not been highly tuned, and work surprisingly well regardless of task.
Revised: These constants have not been highly tuned, and work surprisingly well regardless of task.

Original: Ensuring a minimum of 1 means that under-used nodes are never completely ignored during a merge.
Revised: Ensuring a minimum of 1 means that under-used nodes are never completely ignored during a merge.

Original: • To adjust for the fact that rewards are not consistently scaled across tasks, we maintain a max reward constant, set to the absolute value of an observed reward that exceeds the current max reward .
Revised: • To adjust for the fact that rewards are not consistently scaled across tasks, we maintain a max reward constant, set to the absolute value of an observed reward that exceeds the current max reward .

Original: This value is multiplied by .
Revised: This value is multiplied by .

Original: 999 every timestep.
Revised: 999 every timestep.

Original: All rewards are scaled as 10 ∗ reward/max reward , to scale their values between [-10, 10].
Revised: All rewards are scaled as 10 ∗ reward/max reward , to scale their values between [-10, 10].

Original: • The critic network’s weights are initialized according to a normal distribution with meanand standard deviation 0 .
Revised: • The critic network’s weights are initialized according to a normal distribution with meanand standard deviation 0 .

Original: 001 /in features .
Revised: 001 /in features .

Original: In other words, the values are very close to 0.
Revised: In other words, the values are very close to 0.

Original: • To adjust for the fact that the very ﬁrst node’s anchor is initialized randomly, and thus not effective as a baseline, we have an edge-case way to initiate the promotion of a node: when it is used more than 10,000 times.
Revised: • To adjust for the fact that the very ﬁrst node’s anchor is initialized randomly, and thus not effective as a baseline, we have an edge-case way to initiate the promotion of a node: when it is used more than 10,000 times.

Original: (Usage count 2 above.)
Revised: (Usage count 2 above.)

Original: • We take a leaf out of IMPALA’s book and, since our critics are updated asychronously, every node has a duplicate critic that is what gets trained, and then its weights are copied into the shared critic that is used during data collection.
Revised: • We take a leaf out of IMPALA’s book and, since our critics are updated asychronously, every node has a duplicate critic that is what gets trained, and then its weights are copied into the shared critic that is used during data collection.

Original: This does mean critic weights change mid-data collection, but not mid-training.
Revised: This does mean critic weights change mid-data collection, but not mid-training.

Original: • Every training step we update the non-leaf nodes’ policies to be the weighted average of their children’s.
Revised: • Every training step we update the non-leaf nodes’ policies to be the weighted average of their children’s.

Original: These policies are used for two purposes: (1) if the node is cloned, its policy will end up as the policy of a leaf, and thus used, and (2) the policies are used to determine which nodes to merge, so a reasonably meaningful one is beneﬁcial.
Revised: These policies are used for two purposes: (1) if the node is cloned, its policy will end up as the policy of a leaf, and thus used, and (2) the policies are used to determine which nodes to merge, so a reasonably meaningful one is beneﬁcial.

Original: These aspects are primarily important in trees with more than two layers.
Revised: These aspects are primarily important in trees with more than two layers.

Original: • Periodically (every 100 updates) we send to each non-leaf node a batch of entries from each of its children.
Revised: • Periodically (every 100 updates) we send to each non-leaf node a batch of entries from each of its children.

Original: This is in case one of the children has been getting activated to the exclusion of the others, ensuring the parent still represents its other children.
Revised: This is in case one of the children has been getting activated to the exclusion of the others, ensuring the parent still represents its other children

Original: • When bootstrapping the estimated value for the last entry in the replay buffer to compute the discounted rewards, we found it helpful to multiply that value by 0.9.
Revised: • When bootstrapping the estimated value for the last entry in the replay buffer to compute the discounted rewards, we found it helpful to multiply that value by 0.9.

Original: This is so if the estimate is wrong, it doesn’t perpetuate itself via the discounted rewards entered into the replay buffer.
Revised: This is so if the estimate is wrong, it doesn’t perpetuate itself via the discounted rewards entered into the replay buffer.

Original: A. SANE E XPERIMENT D ETAILS
Revised: A. SANE E XPERIMENT D ETAILS

Original: The majority of SANE’s constants are common between experiments: 
Revised: Many of SANE’s constants are common between experiments: Description

Original: [Table]
Revised: Value

Original:  rate 1eReplay buffer sizeCritic train epochs Merge ratio 3/
Revised: Policy learning rate 1eCritic learning rate 1eReplay buffer size

Original: Num parallel envs

Original: Timesteps between updates ( T )

Original: Min usages for promotion k m
Revised: Min usages for promotion k m

Original: MiniGrid constants:
Revised: MiniGrid constants:

Original: Description
Revised: Description

Original: Value
Revised: Value

Revised: Num parallel envs

Revised: Timesteps between updates ( T )

Revised: Error scale α 0.

Revised: Merge ratio

Revised: Critic train epochs

Original: Replay batch size ( k )
Revised: Replay batch size ( k )

Original: Reward discount γ .
Revised: Reward discount γ .

Original: Epsilon (cid:15) 0.
Revised: Epsilon (cid:15) 0.

Original: MNIST constants:
Revised: MNIST constants:

Original: Description
Revised: Description

Original: Value
Revised: Value

Revised: Num parallel envs

Revised: Timesteps between updates ( T )

Revised: Error scale α 1.

Revised: Merge ratio 3/

Revised: Critic train epochs

Original: Replay batch size ( k )
Revised: Replay batch size ( k )

Original: Reward discount γ
Revised: Reward discount γ

Original: Epsilon (cid:15) 0.
Revised: Epsilon (cid:15) 0.

Original: MNIST gets divided into more batches per training cycle, MNIST’s discount factor must be 0 because the images presented to the agent have no correlation to each other in time, and MNIST gets a higher exploration rate.
Revised: MNIST gets divided into more batches per training cycle, MNIST’s discount factor must be 0 because the images presented to the agent have no correlation to each other in time, and MNIST gets a higher exploration rate.

Original: A. B ASELINE I MPLEMENTATION D ETAILS
Revised: A. B ASELINE I MPLEMENTATION D ETAILS

Original: For IMPALA we used the Torchbeast implementation (K¨uttler et al., 2019).
Revised: For IMPALA we used the Torchbeast implementation (K ¨ uttler et al., 2019).

Original: We built CLEAR on top of this base.
Revised: We built CLEAR on top of this base.

Original: We used the same constants for both baselines.
Revised: We used the same constants for both baselines, except IMPALA used an entropy cost of 0 . 0001 .

Original: Code is available in continual rl/policies/impala/torchbeast/monobeast.py .
Revised: Code is available in continual rl/policies/impala/torchbeast/monobeast.py .

Original: torchbeast’s default values are available in continual rl/policies/impala/impala policy config.py
Revised: torchbeast’s default values are available in continual rl/policies/impala/impala policy config.py

Original: Listed here are the ones we changed:
Revised: Listed here are the ones we changed:

Original: MiniGrid constants
Revised: MiniGrid constants

Original: Description
Revised: Description

Original: Value
Revised: Value

Original: Unroll length ( k )
Revised: Unroll length ( k )

Original: Entropy cost 0.
Revised: Entropy cost 0.

Original: Learning rate 0.
Revised: Learning rate 0.

Original: Epsilon (cid:15) 0.

Original: Num learner threads
Revised: Num learner threads

Original: Discounting 0.
Revised: Discounting 0.

Original: MNIST constants
Revised: MNIST constants

Original: Description
Revised: Description

Original: Value
Revised: Value

Original: Unroll length ( k )
Revised: Unroll length ( k )

Original: Entropy cost 0.
Revised: Entropy cost 0.

Original: Learning rate 0.
Revised: Learning rate 0.

Original: Epsilon (cid:15) 0.

Original: Num learner threads
Revised: Num learner threads

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original:  N ETWORK S PECIFICATIONS
Revised: A. N ETWORK S PECIFICATIONS

Original: To simplify things, we adopt the convention of describing 2d convolutional layers with
Revised: To simplify things, we adopt the convention of describing 2d convolutional layers with

Original: Conv ( c, w, s ) , indicating there are c output channels, window size w × w , and stride s .
Revised: Conv ( c, w, s ) , indicating there are c output channels, window size w × w , and stride s .

Original: Similarly, we denote max pooling over a window size of w × w with MaxPool ( w ) .
Revised: Similarly, we denote max pooling over a window size of w × w with MaxPool ( w ) .

Original: Finally, we denote a linear layer with output c channels with Lin ( c ) .
Revised: Finally, we denote a linear layer with output c channels with Lin ( c ) .

Original: In all cases the non-linearity used is ReLU, and for simplicity remains implied.
Revised: In all cases the non-linearity used is ReLU, and for simplicity remains implied.

Original: A.6. B ASE N ETWORKS
Revised: A.6. B ASE N ETWORKS

Original: For MiniGrid the base encoder, shared by SANE, CLEAR, and IMPALA is
Revised: For MiniGrid the base encoder, shared by SANE, CLEAR, and IMPALA is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: For MNIST the base encoder is:
Revised: For MNIST the base encoder is:

Original: [Equation]
Revised: [Equation]

Original: For CLEAR and IMPALA, the output of these encoders is concatenated with a one-hot of the last action and the last reward, and projected using a linear layer to the ﬁnal output size: the total number of actions (10 for MNIST, 7 for MiniGrid).
Revised: For CLEAR and IMPALA, the output of these encoders is concatenated with a one-hot of the last action and the last reward, and projected using a linear layer to the ﬁnal output size: the total number of actions (10 for MNIST, 7 for MiniGrid).

Original: For SANE, the encoder is followed by
Revised: For SANE, the encoder is followed by

Original: [Equation]
Revised: [Equation]

Original: Where the the 2 output values are the expected value v and the uncertainty u for that node.
Revised: Where the the 2 output values are the expected value v and the uncertainty u for that node.

Original: This gives a total of about 75,000 parameters for MNIST and around 35,000 for MiniGrid.
Revised: This gives a total of about 75,000 parameters for MNIST and around 35,000 for MiniGrid.

Original: A.6. I NCREASED P ARAMETER N ETWORKS
Revised: A.6. I NCREASED P ARAMETER N ETWORKS

Original: A number of iterations were tried for these larger networks, particularly with emphasis on maintaining certain bottleneck layer dimensions, though we do not claim to have spanned the whole space of options.
Revised: A number of iterations were tried for these larger networks, particularly with emphasis on maintaining certain bottleneck layer dimensions, though we do not claim to have spanned the whole space of options.

Original: These are what were used in the paper:
Revised: These are what were used in the paper:

Original: For MiniGrid, giving around 1.2M parameters:
Revised: For MiniGrid, giving around 1.2M parameters:

Original: [Equation]
Revised: [Equation]

Revised: (1024 , 1 , 1) → MaxPool (2) → Conv (32 , 2 , 1) → Conv (64 , 2 , 1) → Flatten → Lin (32) (7)

Original: For MNIST, giving around 5.5M parameters:
Revised: For MNIST, giving around 5.5M parameters:

Original: Conv
Revised: Conv

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] [Equation]
Revised: 24 , 5 , 1) → Conv (2048 , 1 , 1) → Conv (2048 , 1 , 1) → Conv (24 , 1 , 1) → MaxPool (2) → Conv (48 , 5 , 1) → Conv (1024 , 1 , 1) → Conv (1024 , 1 , 1) → Conv (48 , 1 , 1) → MaxPool (2) → Flatten → Lin (32) (8)

Original: [Equation]
Revised: [Figure]

Original: [Equation] [Equation] [Equation] → Conv (48 , 5 , 1) → Conv
Revised: In Figure 9 we present a comparison of CLEAR runs, varying the fraction of each training batch that is sampled from the replay buffer.

Original: [Equation] → Conv (1024 , 1 , 1) → Conv (48 , 1 , 1) → MaxPool (2) →
Revised: We can see that the higher ratios do indeed both learn more slowly (for instance digit 2), and remember previous tasks longer.

Original: [Table]
Revised: Higher ratios did not reliably learn the digits, so they were omitted from the analysis. From this we see that r=0.88 does the best, so that is what we use in the comparison in Figure 4. We additionally ran early stopping (stopping after exceeding a threshold of 0 . 97 ), but it exhibited no recall on past experiments, so we omitted it to reduce clutter in the graph.

Original: → Lin (32) (8)
Revised: [Figure] In Figure 10 we see our comparison for Minigrid, with the same method as used for MNIST. Early stopping is done when the value exceeds (0.78, 0.95, 0.92) respectively, selected by observing prior runs. Note replay ratio of 0.88 performs poorly in the second task and does not succeed at all at the third task, allowing for the superior recall of the ﬁrst task.

Revised: Early stopping and a replay ratio of 0.75 perform comparably; we select early stopping for our comparison due to better asymptotic performance in 2 Room.

Revised: Note that, by chance, every experiment type (including SANE) had one run fail to learn task 3 within the allotted time, and was thus re-run.