Original: C
Revised: C

Original: ONTINUAL L EARNING
Revised: ONTINUAL L EARNING

Original: VIA
Revised: VIA

Original: E XPLICIT S TRUCTURE
Revised: E XPLICIT S TRUCTURE

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite recent advances in deep learning, neural networks suffer catastrophic forgetting when tasks are learned sequentially.
Revised: Despite recent advances in deep learning, neural networks suffer catastrophic forgetting when tasks are learned sequentially.

Original: We propose a conceptually simple and general framework for continual learning, where structure optimization is considered explicitly during learning.
Revised: We propose a conceptually simple and general framework for continual learning, where structure optimization is considered explicitly during learning.

Original: We implement this idea by separating the structure and parameter learning.
Revised: We implement this idea by separating the structure and parameter learning.

Original: During structure learning, the model optimizes for the best structure for the current task.
Revised: During structure learning, the model optimizes for the best structure for the current task.

Original: The model learns when to reuse or modify structure from previous tasks, or create new ones when necessary.
Revised: The model learns when to reuse or modify structure from previous tasks, or create new ones when necessary.

Original: The model parameters are then estimated with the optimal structure.
Revised: The model parameters are then estimated with the optimal structure.

Original: Empirically, we found that our approach leads to sensible structures when learning multiple tasks continuously.
Revised: Empirically, we found that our approach leads to sensible structures when learning multiple tasks continuously.

Original: Additionally, catastrophic forgetting is also largely alleviated from explicit learning of structures.
Revised: Additionally, catastrophic forgetting is also largely alleviated from explicit learning of structures.

Original: Our method also outperforms all other baselines on the Visual Domain Decathlon Dataset with reasonable parameter overhead.
Revised: Our method also outperforms all other baselines on the permuted MNIST dataset in continual learning setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning different tasks continuously is a common and practical scenario that happens all through the course of human learning.
Revised: Learning different tasks continuously is a common and practical scenario that happens all through the course of human learning.

Original: The learning of new skills from new tasks usually does not have negative impact on the previously learned tasks.
Revised: The learning of new skills from new tasks usually does not have negative impact on the previously learned tasks.

Original: Furthermore, with learning multiple tasks that are highly related, it often helps to advance all related skills.
Revised: Furthermore, with learning multiple tasks that are highly related, it often helps to advance all related skills.

Original: However, this is commonly not the case in current deep learning models.
Revised: However, this is commonly not the case in current deep learning models.

Original: When presented a sequence of learning tasks, the model experiences so called “catastrophic forgetting” problem (McCloskey & Cohen, 1989; Ratcliff, 1990), where the model “forgets” the previous learned task while learning the new task.
Revised: When presented a sequence of learning tasks, the model experiences so called “catastrophic forgetting” problem (McCloskey & Cohen, 1989; Ratcliff, 1990), where the model “forgets” the previous learned task while learning the new task.

Original: This is an interesting phenomenon that has attracted lots of research recently.
Revised: This is an interesting phenomenon that has attracted lots of research recently.

Original: Many efforts have been tried to overcome catastrophic forgetting.
Revised: Many efforts have been tried to overcome catastrophic forgetting.

Original: Common approaches such as
Revised: Common approaches such as

Original: Elastic weight consolidation (EWC Kirkpatrick et al., 2017) and synaptic intelligence (Zenke et al., 2017) focuses on alleviating catastrophic forgetting by applying constraints on the update of the parameters.
Revised: Elastic weight consolidation (EWC Kirkpatrick et al., 2017) and synaptic intelligence (Zenke et al., 2017) focuses on alleviating catastrophic forgetting by applying constraints on the update of the parameters.

Original: However, forgetting is still non-negligible with these approaches, especially when the number of tasks increases.
Revised: However, forgetting is still non-negligible with these approaches, especially when the number of tasks increases.

Original: Forgetting could also be alleviated with memory-based methods, where certain information regarding learned tasks are stored to help retaining the performance of the learned tasks (see Lopez-Paz et al., 2017; Sener & Savarese, 2018, for example).
Revised: Forgetting could also be alleviated with memory-based methods, where certain information regarding learned tasks are stored to help retaining the performance of the learned tasks (see Lopez-Paz et al., 2017; Sener & Savarese, 2018, for example).

Original: Additionally, there are also methods (Mallya & Lazebnik, 2018;
Revised: Additionally, there are also methods (Mallya & Lazebnik, 2018;

Original: Rebufﬁ et al., 2017a; 2018; Mancini et al., 2018) that could learn multiple domains and completely avoid forgetting by adding a small portion of parameters while the original weights is kept ﬁxed.
Revised: Rebufﬁ et al., 2017a; 2018; Mancini et al., 2018) that could learn multiple domains and completely avoid forgetting by adding a small portion of parameters while the original weights is kept ﬁxed.

Original: However, these kind of models rely on a strong base network and knowledge can only be transferred fromthe one task to another.
Revised: However, these kind of models rely on a strong base network and knowledge can only be transferred from one task to another.

Original: Most of the current approaches in continual learning employ a model with a ﬁxed structure, and apply the same model to learn all tasks.
Revised: Most of the current approaches in continual learning employ a model with a ﬁxed structure, and apply the same model to learn all tasks.

Original: We believe that a more intuitive and sensible approach is to learn task speciﬁc model structure explicitly along with learning of the model parameters 1 .
Revised: We believe that a more intuitive and sensible approach is to learn task speciﬁc model structure explicitly along with learning of the model parameters 1 .

Original: Since each task may require different architecture.
Revised: Since each task may require different architecture.

Original: In case two tasks are not quite relevant, it may not make much sense to employ the same structure for learning.
Revised: In case two tasks are not quite relevant, it may not make much sense to employ the same structure for learning.

Original: For example, in case of learning digit and face recognition models, the lower level features required is likely to be drastically different, and similarly for the required structure.
Revised: For example, in case of learning digit and face recognition models, the lower level features required is likely to be drastically different, and similarly for the required structure.

Original: Forcing the same structure for these tasks is likely to prevent
Revised: Forcing the same structure for these tasks is likely to prevent

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: [Figure]
Revised: [Figure]

Original: S S
Revised: S S

Original: S
Revised: S

Original: T B T A
Revised: T B T A

Original: S
Revised: S

Original: S S
Revised: S S

Original: S
Revised: S

Original: T B T A
Revised: T B T A

Original: S
Revised: S

Original: T B T A
Revised: T B T A

Original: S
Revised: S

Original: S S
Revised: S S

Original: S 4 ,
Revised: S 4 ,

Original: [Figure]
Revised: [Figure]

Original: sharing as very less information is relevant between the tasks.
Revised: sharing, since very less information is relevant between the tasks.

Original: On the other hand, if tasks were allowed to use different structure, it has the potential to encourage information sharing.
Revised: On the other hand, if tasks were allowed to use different structure, it has the potential to encourage information sharing.

Original: Since now the irrelevant part can be handled by the new/different structure.
Revised: This is because now the irrelevant part can be handled by the new/different structure.

Original: Therefore, in this work we propose a framework that explicitly separate the learning of model structure and model parameters.
Revised: Therefore, in this work we propose a framework that explicitly separate the learning of model structure and model parameters.

Original: In particular, we employ architecture search for each of the sequential tasks to ﬁnd the optimal structure for the current task.
Revised: In particular, we employ architecture search for each of the sequential tasks to ﬁnd the optimal structure for the current task.

Original: The search considers various options, such as share previous layer’s parameter, spawn new parameters, and so on.
Revised: The search considers various options, such as share previous layer’s parameter, spawn new parameters, and so on.

Original: The model parameters are then learned correspondingly.
Revised: The model parameters are then learned correspondingly.

Original: We found that 1) qualitatively, the learned structure is sensible – similar tasks tends to share more parameters and structure, whereas distant tasks share less; 2) quantitatively, separating the structure and parameter learning signiﬁcantly reduced catastrophic forgetting.
Revised: We found that 1) qualitatively, the learned structure is sensible – similar tasks tends to share more parameters and structure, whereas distant tasks share less; 2) quantitatively, separating the structure and parameter learning signiﬁcantly reduced catastrophic forgetting.

Original: Additionally, we propose a new approach for doing continuous learning with architecture search.
Revised: Additionally, we propose a new approach for doing continuous learning with architecture search.

Original:  E XPLICIT S TRUCTURE L EARNING
Revised:  E XPLICIT S TRUCTURE L EARNING

Original: The continual learning task that we are concerned with in this work is deﬁned as follows.
Revised: The continual learning task that we are concerned with in this work is deﬁned as follows.

Original: N tasks
Revised: N tasks

Original: T 1 , T 2 , . .
Revised: T 1 , T 2 , . .

Original: , T N forms a task sequence T =
Revised: , T N forms a task sequence T =

Original: ( T 1 , T 2 , ..., T N ) .
Revised: ( T 1 , T 2 , ..., T N ) .

Original: Each task T i is composed of dataset
Revised: Each task T i is composed of dataset

Original: D i = { ( x ( i ) 1 , y
Revised: D i = { ( x ( i ) 1 , y

Original: ( i ) 1 ) , ( x ( i ) 2 , y ( i ) 2 ) , . . .
Revised: ( i ) 1 ) , ( x ( i ) 2 , y ( i ) 2 ) , . . .

Original: , ( x ( i ) N Ti , y ( i ) N Ti ) } , where each dataset is of size N T
Revised: , ( x ( i ) N Ti , y ( i ) N Ti ) } , where each dataset is of size N T

Original: i .
Revised: i .

Original: The model gets to observe tasks from 1 to N sequentially.
Revised: The model gets to observe tasks from 1 to N sequentially.

Original: After the model ﬁnished learning on task T i , the model can no longer access this task, i.e. all data from D i will not be available when learning tasks
Revised: After the model ﬁnished learning on task T i , the model can no longer access this task, i.e. all data from D i will not be available when learning tasks

Original: T i +1 to T N .
Revised: T i +1 to T N .

Original: Ideally, we would like to minimize the following loss function in this continuous learning setting
Revised: Ideally, we would like to minimize the following loss function in this continuous learning setting

Original: [Equation]
Revised: [Equation]

Original: where f θ is the model and (cid:96) i is the loss function for task T i .
Revised: where f θ is the model and (cid:96) i is the loss function for task T i .

Original: However, since we do not have access to all dataset at the same time, the above loss in equation 1 is very hard to minimize.
Revised: However, since we do not have access to all dataset at the same time, the above loss in equation 1 is very hard to minimize.

Original: In case we ignore the access issue and train each task separately, this will result in catastrophic forgetting.
Revised: In case we ignore the access issue and train each task separately, this will result in catastrophic forgetting.

Original: Additionally, the above loss deﬁnition clearly does not account for structure changes explicitly in the formulation.
Revised: Additionally, the above loss deﬁnition clearly does not account for structure changes explicitly in the formulation.

Original: As motivated earlier, we believe explicit learning of structure is important for continuous learning.
Revised: As motivated earlier, we believe explicit learning of structure is important for continuous learning.

Original: The key intuition for explicit learning structures for new tasks is that in case there is high dissimilarity between the current and previously seen tasks, optimizing model parameter may inevitably cause forgetting.
Revised: The key intuition for explicit learning structures for new tasks is that in case there is high dissimilarity between the current and previously seen tasks, optimizing model parameter may inevitably cause forgetting.

Original: This is because the model isbeen asked to function very differently, and it is very unlikely that the new optimal parameters will also be a good solution for the previous tasks.
Revised: This is because the model is asked to function very differently, and it is very unlikely that the new optimal parameters will also be a good solution for the previous tasks.

Original: Even in case where the tasks are similar, it may not be ideal to share all of the structure, as there might be ﬁne grained details in tasks that make the part of the model focus on extracting different types of representations.
Revised: Even in case where the tasks are similar, it may not be ideal to share all of the structure, as there might be ﬁne grained details in tasks that make the part of the model focus on extracting different types of representations.

Original: Therefore, we introduce here s i ( θ ) to indicate the structure for task T i .
Revised: Therefore, we introduce here s i ( θ ) to indicate the structure for task T i .

Original: The updated loss for individual task is then
Revised: The updated loss for individual task is then

Original: [Equation]
Revised: [Equation]

Original: Now the structure is explicitly taken into consideration when learning all the tasks.
Revised: Now the structure is explicitly taken into consideration when learning all the tasks.

Original: When optimizing the updated loss in equation 3, one needs to determine the optimal parameter based on the structure s i .
Revised: When optimizing the updated loss in equation 3, one needs to determine the optimal parameter based on the structure s i .

Original: This loss can be viewed in two ways.
Revised: This loss can be viewed in two ways.

Original: One can interpret the above loss as selecting a task speciﬁc network from a ‘super network’ that has parameter θ using s i , or for each task we train a new model with parameter s i ( θ ) .
Revised: One can interpret the above loss as selecting a task speciﬁc network from a ‘super network’ that has parameter θ using s i , or for each task we train a new model with parameter s i ( θ ) .

Original: There is a subtle difference between this two views.
Revised: There is a subtle difference between this two views.

Original: The former one has an constraint on the total model size, where as the latter one does not.
Revised: The former one has an constraint on the total model size, where as the latter one does not.

Original: So in worst case scenario, the model size will grow linearly as we increase the number of tasks.
Revised: So in worst case scenario, the model size will grow linearly as we increase the number of tasks.

Original: This would lead to a trivial solution – training completely different models for different tasks and is no longer continuous learning!
Revised: This would lead to a trivial solution – training completely different models for different tasks and is no longer continuous learning!

Original: To address this problem, we propose the following loss
Revised: To address this problem, we propose the following loss

Original: [Equation]
Revised: [Equation]

Original: where β i > 0 , λ i ≥ 0 , R si and R pi indicate regularizer for structure and parameter, respectively.
Revised: where β i > 0 , λ i ≥ 0 , R si and R pi indicate regularizer for structure and parameter, respectively.

Revised: For instance, one can use (cid:96) 2 regularization for R pi when optimizing model parameters, and R si can be as simple as the ( log ) number of parameters.

Original: In this way, the total number of parameters are upper bounded, and the degenerate case is thus avoided.
Revised: In this way, the total number of parameters are upper bounded, and the degenerate case is thus avoided.

Original:  O UR I MPLEMENTATION
Revised:  O UR I MPLEMENTATION

Original: It is a challenging problem to optimize the loss described in equation 4, since it involves explicit optimization of the structure of the model.
Revised: It is a challenging problem to optimize the loss described in equation 4, since it involves explicit optimization of the structure of the model.

Original: In our implementation, we choose to separate this optimization problem to two steps: structure optimization and model parameter learning.
Revised: In our implementation, we choose to separate this optimization problem to two steps: structure optimization and model parameter learning.

Original: In particular, we employ neural architecture search to deal with structure optimization, and then gradient based method for learning model parameters after the structure is ﬁxed.
Revised: In particular, we employ neural architecture search to deal with structure optimization, and then gradient based method for learning model parameters after the structure is ﬁxed.

Original: We explain our implementation in detail in the following section.
Revised: We explain our implementation in detail in the following section.

Original: 3. S TRUCTURE O PTIMIZATION
Revised: 3. S TRUCTURE O PTIMIZATION

Original: We employ neural architecture search for structure optimization.
Revised: We employ neural architecture search for structure optimization.

Original: Before we move on to further details, we make a further simpliﬁcation, where we assume that one already have in mind a global structure that may work for all tasks, and we are only selecting connectivity pattern between layers and their corresponding operator.
Revised: Before we move on to further details, we make a further simpliﬁcation, where we assume that one already have in mind a global structure that may work for all tasks, and we are only selecting connectivity pattern between layers and their corresponding operator.

Original: It is straight forward to adapt this to more complicated cases, we make the simpliﬁcation because: 1) it is common in a multi-task continual learning scenario that one has some rough clue regarding the overall model structure; 2) this simpliﬁes the optimization problem signiﬁcantly.
Revised: It is straight forward to adapt this to more complicated cases, we make the simpliﬁcation because: 1) it is common in a multi-task continual learning scenario that one has some rough clue regarding the overall model structure; 2) this simpliﬁes the optimization problem signiﬁcantly.

Original: Let’s deﬁne a certain network with L shareable layers and one task-speciﬁc layer (i.e. last layer) for each task.
Revised: Let’s deﬁne a certain network with L shareable layers and one task-speciﬁc layer (i.e. last layer) for each task.

Original: A super network S is maintained so that all the new task-speciﬁc layers and new shareable layers will be stored into S .
Revised: A super network S is maintained so that all the new task-speciﬁc layers and new shareable layers will be stored into S .

Original: The goal of search is trying to ﬁnd out the optimal choice for each of the L layers, given the current task data D i and all the shareable layer’s weights stored in S .
Revised: The goal of search is trying to ﬁnd out the optimal choice for each of the L layers, given the current task data D i and all the shareable layer’s weights stored in S .

Original: The candidate choices for each layer could be “reuse”, “adaptation” and “new”.
Revised: The candidate choices for each layer could be “reuse”, “adaptation” and “new”.

Original: The reuse choice will make new task use the same parameter as the previous task.
Revised: The reuse choice will make new task use the same parameter as the previous task.

Original: The adaptation option adds a small parameter overhead that trains an additive function to the original layer output.
Revised: The adaptation option adds a small parameter overhead that trains an additive function to the original layer output.

Original: The new operator will spawn new parameters ofexactly the size of the current layer parameters. Here, we denote the size of the l th layer in super network S as |S l | . The total number of choices in the l th layer C l is 2 |S l | + 1 , because we will have
Revised: The new operator will spawn new parameters of

Original: S(a) (c)
Revised: S(a) (c)

Original: S S
Revised: S S

Original: S S
Revised: S S

Original: S
Revised: S

Original: S 4 S
Revised: S 4 S

Original: T (b)
Revised: T (b)

Original: S S
Revised: S S

Original: T k
Revised: T k

Original: S S
Revised: S S

Original: S(d)
Revised: S(d)

Original: S
Revised: S

Original: S S
Revised: S S

Original: S
Revised: S

Original: S 4 S
Revised: S 4 S

Original: T
Revised: T

Original: S S
Revised: S S

Original: S
Revised: S

Original: S 1 S
Revised: S 1 S

Original: S
Revised: S

Original: S 2 S S 2 S
Revised: S 2 S S 2 S

Original: S
Revised: S

Original: S 4 S S 4 S
Revised: S 4 S S 4 S

Original: S 4 S
Revised: S 4 S

Original: T k α
Revised: T k α

Original: S 3 S
Revised: S 3 S

Original: S, α 2 ,, , , , α,, ,, ,,, , ,
Revised: S, α 2 ,, , , , α,, ,, ,,, , ,

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  |S l | ”reuse”, |S l | ”adaptation” and 1 ”new”.
Revised: exactly the size of the current layer parameters. Here, we denote the size of the l th layer in super network S as |S l | . The total number of choices in the l th layer C l is 2 |S l | + 1 , because we will have |S l | ”reuse”, |S l | ”adaptation” and 1 ”new”.

Original: Thus, the total search space is (cid:81)
Revised: Thus, the total search space is (cid:81)

Original: Ll C l .
Revised: Ll C l .

Original: One potential issue here is that, in worst case, the search space may grow exponentially with respect to the number of tasks.
Revised: One potential issue here is that, in worst case, the search space may grow exponentially with respect to the number of tasks.

Original: One way of dealing with this is to limit the total number of possible choices, and make use a priority queue for managing the options.
Revised: One way of dealing with this is to limit the total number of possible choices, and make use a priority queue for managing the options.

Original: However, we do not ﬁnd this necessary in all of our experiments.
Revised: However, we do not ﬁnd this necessary in all of our experiments.

Original: Similar to DARTS (Liu et al., 2018), to make the search space continuous, we relax the categorical choice of the l th layer as a softmax over all possible C l choices:
Revised: Similar to DARTS (Liu et al., 2018), to make the search space continuous, we relax the categorical choice of the l th layer as a softmax over all possible C l choices:

Original: [Equation]
Revised: [Equation]

Original: Here, the vector α l of dimension C l is the architecture weights that are used for mixing the choices for each sharealbe layer.
Revised: Here, the vector α l of dimension C l is the architecture weights that are used for mixing the choices for each sharable layer.

Original: And g lc here is the operator for the choice c at layer l which is expressed as:
Revised: And g lc here is the operator for the choice c at layer l which is expressed as:

Original: [Equation]
Revised: [Equation]

Original: Here, γ is the adapter operator and o is the new operator training from scratch.
Revised: Here, γ is the adapter operator and o is the new operator training from scratch.

Original: After this relaxation, the task of discrete search become optimizing a set of continuous weights α = (cid:8) α l (cid:9) .
Revised: After this relaxation, the task of discrete search become optimizing a set of continuous weights α = (cid:8) α l (cid:9) .

Original: After ﬁnishing the searching, the optimal architecture could be obtained by taking the index with the largest weight α lc for each layer l , i.e. c l = arg max α l .
Revised: After ﬁnishing the searching, the optimal architecture could be obtained by taking the index with the largest weight α lc for each layer l , i.e. c l = arg max α l .

Original: Adopting the training strategy from DARTS.
Revised: Adopting the training strategy from DARTS.

Original: We use validation loss L val to update the architecture weights α , while the operator weights are optimized by the training loss L train .
Revised: We use validation loss L val to update the architecture weights α , while the operator weights are optimized by the training loss L train .

Original: The architecture weights and operator weights are updated alternately during the search process.
Revised: The architecture weights and operator weights are updated alternately during the search process.

Original: Because it is a nested bi-level optimization problem, the original DARTS provide a second-order approximation for more accurate optimization.
Revised: Because it is a nested bi-level optimization problem, the original DARTS provide a second-order approximation for more accurate optimization.

Original: In this work, we ﬁnd it is sufﬁcient to use the simple alternately update approach, which was referred as the ﬁrst-order approximation in (Liu et al., 2018).
Revised: In this work, we ﬁnd it is sufﬁcient to use the simple alternately update approach, which was referred as the ﬁrst-order approximation in (Liu et al., 2018).

Original: To make it clear how “reuse”, “adaptation” and “new” operator works, we walk through a concrete example in the following.
Revised: To make it clear how “reuse”, “adaptation” and “new” operator works, we walk through a concrete example in the following.

Original: Let us take a convolutional neural network (CNN) with all the layers using 3x3 kernel size as an example.
Revised: Let us take a convolutional neural network (CNN) with all the layers using 3x3 kernel size as an example.

Original: The choice of “reuse” is just using the existing weight and keep it ﬁxed during learning, thus there is no additional parameter cost.
Revised: The choice of “reuse” is just using the existing weight and keep it ﬁxed during learning, thus there is no additional parameter cost.

Original: For “adaptation”, it could be a 1x conv layer added to the original 3x3 conv layer in parallel (i.e. similar to the adaptor used in paper (Rebufﬁ et al., 2017a)).
Revised: For “adaptation”, it could be a 1x conv layer added to the original 3x3 conv layer in parallel (i.e. similar to the adaptor used in paper (Rebufﬁ et al., 2017a)).

Original: During training, the weight of the original 3x3 conv is kept ﬁxed, while the parameters of the 1x1 conv adapter is modiﬁed.
Revised: During training, the weight of the original 3x3 conv is kept ﬁxed, while the parameters of the 1x1 conv adapter is modiﬁed.

Original: In this case, the additional parameter cost is only 1/9 of the original parameter size.
Revised: In this case, the additional parameter cost is only 1/9 of the original parameter size.

Original: In case of “new”, it is literally start with a new operator that initialized randomly and train from scratch.
Revised: In case of “new”, it is literally start with a new operator that initialized randomly and train from scratch.

Original: We make use of the loss function L val to implement the regularizerfor the structure R si ( s i ) .
Revised: We make use of the loss function L val to implement the regularizer R si ( s i ) .

Original: The value of the loss function is set proportional to the product of the additional parameter size z lc and its corresponding weight α lc .
Revised: The value of the regularizer is set proportional to the product of the additional parameter size z lc and its corresponding weight α lc (i.e. R si ( s i ) = (cid:80) c,l α lc z lc ).

Original: The architecture weights α is optimized in terms of both accuracy and parameter efﬁciency at the same time.
Revised: The architecture weights α is optimized in terms of both accuracy and parameter efﬁciency at the same time.

Original: 3. P ARAMETER O PTIMIZATION
Revised: 3. P ARAMETER O PTIMIZATION

Original: After we get the optimal choices for each layer from the search procedure, we retrain the optimal architecture on the current task.
Revised: After we get the optimal choices for each layer from the search procedure, we retrain the optimal architecture on the current task.

Original: There are two strategies to deal with “reuse”, we can either ﬁx it unchanged during retraining just as in search, or we can tune it with some regularization – simple (cid:96) 2 regularization or more sophisticated regularizations like elastic weight consolidation (Kirkpatrick et al., 2017).
Revised: There are two strategies to deal with “reuse”, we can either ﬁx it unchanged during retraining just as in search, or we can tune it with some regularization – simple (cid:96) 2 regularization or more sophisticated regularizations like elastic weight consolidation (Kirkpatrick et al., 2017).

Original: The former strategy could avoid forgetting completely, however it will lose the chance of getting positive backward transfer, which means the learning of new task may help previous tasks performance.
Revised: The former strategy could avoid forgetting completely, however it will lose the chance of getting positive backward transfer, which means the learning of new tasks may help previous tasks’ performance.

Original: When the search process select “reuse” at layer l , it means that the l th layer tends to learn very similar representation as it learned from one of the previous tasks.
Revised: When the search process select “reuse” at layer l , it means that the l th layer tends to learn very similar representation as it learned from one of the previous tasks.

Original: This is an indication of semantic similarity learned at this layer l between the two tasks.
Revised: This is an indication of semantic similarity learned at this layer l between the two tasks.

Original: Thus, we conjecture that tuning the l th layer with some regularization could also beneﬁt the previous tasks, or at least reduce catastrophic forgetting due to the semantic relationships.
Revised: Thus, we conjecture that tuning the l th layer with some regularization could also beneﬁt the previous tasks, or at least reduce catastrophic forgetting due to the semantic relationships.

Original: In the experiment section, we investigate this hypothesis in more detail.
Revised: In the experiment section, we investigate this hypothesis in more detail.

Original: Finally, After retrained on the current task, we need to update/add the created and tuned layers, task-speciﬁc adapters and classiﬁers in the maintained super network.
Revised: Finally, after retrained on the current task, we need to update/add the created and tuned layers, task-speciﬁc adapters and classiﬁers in the maintained super network.

Original: It will be used for model inference and also can be the basis for future architecture search on new tasks.
Revised: It will be used for model inference and also can be the basis for future architecture search on new tasks.

Original:  H OW E XPLICIT S TRUCTURE L EARNING A FFECTS C ONTINUOUS L EARNING
Revised:  H OW E XPLICIT S TRUCTURE L EARNING A FFECTS C ONTINUOUS L EARNING

Original: In this section, we test two main hypothesis that leads to our framework.
Revised: In this section, we test two main hypothesis that leads to our framework.

Original: First, does making structure learning explicit leads to discovery of sensible model architectures for corresponding tasks?
Revised: First, does making structure learning explicit leads to discovery of sensible model architectures for corresponding tasks?

Original: Second, if so, does this leads to better continuous learning?
Revised: Second, if so, does this leads to better continuous learning?

Original: We test these two hypothesis on two datasets: permuated MNIST and the visual domain decathlon dataset (Rebufﬁ et al., 2017a).
Revised: We test these two hypothesis on two datasets: permuted MNIST and the visual domain decathlon dataset (Rebufﬁ et al., 2017a).

Original: The permutated MNIST dataset is a simple image classiﬁcation problem that derived from the MNIST handwritten digit dataset (Yann LeCun, 1998), which is commonly used as benchmark in continual learning literature (Kirkpatrick et al., 2017; Lopez-Paz et al., 2017; Zenke et al., 2017).
Revised: The permuted MNIST dataset is a simple image classiﬁcation problem that derived from the MNIST handwritten digit dataset (Yann LeCun, 1998), which is commonly used as benchmark in continual learning literature (Kirkpatrick et al., 2017; Lopez-Paz et al., 2017; Zenke et al., 2017).

Original: For each task, a unique ﬁxed random permutation is used to shufﬂe the pixels of each image, but the label corresponding to the image is kept ﬁxed.
Revised: For each task, a unique ﬁxed random permutation is used to shufﬂe the pixels of each image, but the label corresponding to the image is kept ﬁxed.

Original: The visual decathlon dataset consists of 10 image classiﬁcation tasks – ImageNet, CIFAR100, Aircraft, DPed, Textures, GTSRB, Omniglot, SVHN, UCF and VGG-Flowers.
Revised: The visual decathlon dataset consists of 10 image classiﬁcation tasks – ImageNet, CIFAR100, Aircraft, DPed, Textures, GTSRB, Omniglot, SVHN, UCF and VGG-Flowers.

Original: The images of all the tasks are resized to lower-edge be 72 pixels.
Revised: The images of all the tasks are resized to lower-edge be 72 pixels.

Original: And the tasks are across multiple domains and dataset sizes are highly imbalanced, which makes it a good candidate to investigate the continual learning problem with potential inter-task transfer.
Revised: And the tasks are across multiple domains and dataset sizes are highly imbalanced, which makes it a good candidate to investigate the continual learning problem with potential inter-task transfer.

Original: For all MNIST experiments, we use a 6-layer fully-connected neural networks with 5 feed-forward layers and the 6 th layer is the task-speciﬁc softmax classiﬁcation layer, as shown in Figure 3(a).
Revised: For all MNIST experiments, we use a 4-layer fully-connected neural networks with 3 feed-forward layers and the 4 th layer is the shared softmax classiﬁcation layer accross all tasks, as shown in Figure 3(a).

Revised: This corresponds to the so called ‘single head’ setup (Farquhar & Gal, 2018).

Revised: We choose to use this setting because for permuted MNIST dataset the semantic of all tasks are all the same, and we think sharing is a more reasonable design choice.

Original: We apply our implementation of the framework for learning 10 permuated MNIST tasks in sequence.
Revised: We apply our implementation of the framework for learning 10 permuted MNIST tasks in sequence.

Original: For simplicity, we only use two choices, “reuse” and “new”, during the structure optimization.
Revised: For simplicity, we only use two choices, “reuse” and “new”, during the structure optimization.

Original: For experiments on visual decathlon dataset, we use a 26-layer ResNet (He et al., 2016) as the base network to learn the ﬁrst task.
Revised: For experiments on visual decathlon dataset, we use a 26-layer ResNet (He et al., 2016) as the base network to learn the ﬁrst task.

Original: This network consists of 3 residual blocks, each output 64, 128,channels.
Revised: This network consists of 3 residual blocks, each output 64, 128,channels.

Original: Each residual block contains 4 residual units, each of which consists of two convolutional layers with 3 × 3 kernels and a skip connection.
Revised: Each residual block contains 4 residual units, each of which consists of two convolutional layers with 3 × 3 kernels and a skip connection.

Original: At the end of each residual block, the feature resolution is halved by average pooling.
Revised: At the end of each residual block, the feature resolution is halved by average pooling.

Original: We adopt three type of choices during the search, i.e.
Revised: We adopt three type of choices during the search, i.e.

Original: “reuse”, “adaptation” and “new”.
Revised: “reuse”, “adaptation” and “new”.

Original: For “adaptation”, a 1 × 1 convolution layer with channels equal to the current layer output channels is added for the corresponding layer.
Revised: For “adaptation”, a 1 × 1 convolution layer with channels equal to the current layer output channels is added for the corresponding layer.

Original: The convolved results are then added back to the convolution output.
Revised: The convolved results are then added back to the convolution output.

Original: During training, the weight of the original layer is kept ﬁxed, while the 1 × 1 layer is adjusted.
Revised: During training, the weight of the original layer is kept ﬁxed, while the 1 × 1 layer is adjusted.

Original: 4. D OES E XPLICIT S TRUCTURE L EARNING L EADS TO S ENSIBLE S TRUCTURES ?
Revised: 4. D OES E XPLICIT S TRUCTURE L EARNING L EADS TO S ENSIBLE S TRUCTURES ?

Original: We ﬁrst experimented on the permutated MNIST dataset.
Revised: We ﬁrst experimented on the permuted MNIST dataset.

Original: One would expect a sensible architecture for these tasks tends to share higher level structures but differs at lower layers.
Revised: One would expect a sensible architecture for these tasks tends to share higher level structures but differs at lower layers.

Original: This is because the high level semantic information of digits is identical throughout, which should imply similar high level structures.
Revised: This is because the high level semantic information of digits is identical throughout, which should imply similar high level structures.

Original: However, since the input pixels are all shufﬂed differently, the lower level features would need to be different.
Revised: However, since the input pixels are all shufﬂed differently, the lower level features would need to be different.

Original: This implies different parameter values should be used in lower layers.
Revised: This implies different parameter values should be used in lower layers.

Original: Interestingly, our experiments show that the structure optimization chooses the same conﬁguration that make brand new ﬁrst layer and reuse all of the followingfour layers in all the 10 tasks (see Figure 3 b).
Revised: Interestingly, our experiments show that the structure optimization chooses the same conﬁguration that make brand new ﬁrst layer and reuse all of the following layers in all the 10 tasks (see Figure 3 b).

Original: This exactly conﬁrms the prediction that we had before, which is a good indication that the structure optimization is working properly.
Revised: This exactly conﬁrms the prediction that we had before, which is a good indication that the structure optimization is working properly.

Original: To further demonstrate that the learned structure indeed make sense, we performed additional experiments.
Revised: To further demonstrate that the learned structure indeed make sense, we performed additional experiments.

Original: Since during the search process on each new task, our model always trying to “new” the ﬁrst layer and “reuse” the other layers.
Revised: Since during the search process on each new task, our model always trying to “new” the ﬁrst layer and “reuse” the other layers.

Original: So, the simple question could be raised here is that what if we use new parameters for other layers?
Revised: So, the simple question could be raised here is that what if we use new parameters for other layers?

Original: Because theﬁve feed-forward layers has similar input and output dimensions, it’s fair to compare the ﬁve setups that the ith layer is “new” and others are “reuse”.
Revised: Because the feed-forward layers has similar input and output dimensions, it’s fair to compare the ﬁve setups that the ith layer is “new” and others are “reuse”.

Original: In the results, we found that using new parameters in the ﬁrst layer is actually the best choice compared with other situations (see Fig.
Revised: In the results, we found that using new parameters in the ﬁrst layer is actually the best choice compared with other situations (see Fig.

Original: 3 d).
Revised: 3 d).

Original: Making the second layer as “new” also results in less forgetting compared with making one of the last three layers as “new”. 
Revised: Making the second layer as “new” also results in less forgetting compared with making one of the last three layers as “new”. [Figure]

Revised: (ﬁx, tune reuse with and without regularization) with SGD and EWC on the average accuracy over the seen tasks.

Revised: d) Ablation experiments of ”new” different layers in terms of average accuracy over the seen tasks.

Original: Next, we run experiments on the visual decathlon dataset, which is a more realistic image dataset.
Revised: Next, we run experiments on the visual decathlon dataset, which is a more realistic image dataset.

Original: To see if the results from structure optimization is consistent with what we have previously, we did two experiments.
Revised: To see if the results from structure optimization is consistent with what we have previously, we did two experiments.

Original: We ﬁrst test on a similary task pair, i.e. ImageNet and CIFAR-100.
Revised: We ﬁrst test on a similary task pair, i.e. ImageNet and CIFAR-100.

Original: In this case, the images are all natural images, although the size of objects are largely different, most of the representation should still be very similar.
Revised: In this case, the images are all natural images, although the size of objects are largely different, most of the representation should still be very similar.

Original: Indeed, as we can see from left of Fig. 5, most of the layers are shared for these two tasks.
Revised: Indeed, as we can see from left of Fig. 5, most of the layers are shared for these two tasks.

Original: Next, we tested on two drastically different tasks, i.e.
Revised: Next, we tested on two drastically different tasks, i.e.

Original: ImageNet and Omniglot.
Revised: ImageNet and Omniglot.

Original: Since these are two very different visual tasks, we would expect minimal sharing in the resulting structure.
Revised: Since these are two very different visual tasks, we would expect minimal sharing in the resulting structure.

Original: As can be see from right of Fig.
Revised: As can be see from right of Fig.

Original: 5, most of the resulting structure leads to use option “new”.
Revised: 5, most of the resulting structure leads to use option “new”.

Original: The above experiments suggest that structure optimization indeed resulting in sensible task speciﬁc structures.
Revised: The above experiments suggest that structure optimization does result in sensible task speciﬁc structures.

Original: The learned task structure tends to share when the semantic representation of corresponding layers are similar, and spawn new parameters when the required information is very different.
Revised: The learned task structure tends to share when the semantic representation of corresponding layers are similar, and spawn new parameters when the required information is very different.

Original: 4. D OES E XPLICIT S TRUCTURE L EARNING A FFECTS F ORGETTING ?
Revised: 4. D OES E XPLICIT S TRUCTURE L EARNING A FFECTS F ORGETTING ?

Original: With ﬁrst hypothesis conﬁrmed, the next natural question is – do sensible structures lead to better continuous learning capability.
Revised: With ﬁrst hypothesis conﬁrmed, the next natural question is – do sensible structures lead to better continuous learning capability.

Original: More speciﬁcally, will explicit structure learning help catastrophic forgetting in any way? 
Revised: More speciﬁcally, will explicit structure learning help catastrophic forgetting in any way? with learning two tasks sequentially.

Revised: The search are based on the super model obtained by training ImageNet as ﬁrst task.

Revised: (a) and (b) shows searched architecture on CIFAR100 and Omniglot task respectively.

Original:  First, we run experiments on the permutated MNIST dataset.
Revised: [Figure] First, we run experiments on the permuted MNIST dataset.

Original: As mentioned in Section 3.2, we have two strategies when the search result gives “reuse” for a certain layer.
Revised: As mentioned in Section 3.2, we have two strategies when the search result gives “reuse” for a certain layer.

Original: We can either ﬁx the reused weight during retraining without the risk forgetting, or we can apply tuning on the ”reused” layers.
Revised: We can either ﬁx the reused weight during retraining without the risk forgetting, or we can apply tuning on the “reused” layers.

Original: As a baseline, we shown that simply updating all the layers with stochastic gradient descent (SGD) results in catastrophic forgetting (see Fig.
Revised: As a baseline, we shown that simply updating all the layers with stochastic gradient descent (SGD) results in catastrophic forgetting (see Fig.

Original: 3 c).
Revised: 3 c).

Original: After trainingtasks, the average accuracy dropped from 97.9% to 63.0%.
Revised: After training 10 tasks, the average accuracy dropped from 97.9% to 63.0%.

Original: With the elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), the forgetting was alleviated to 86% average accuracy overtasks.
Revised: With the elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), the forgetting was alleviated to 96.8% average accuracy over 10 tasks.

Original: For our approach, we found that tuning the “reuse” layers by using simple (cid:96) 2 based regularization on previous task parameter  is safe enough in terms of forgetting and both strategies can keep the overall accuracy as high as training each task individually (see Fig.3). Although our method costs about 1/5 additional parameters (the ﬁrst fc layer) starting from the second task.
Revised: For our approach, we found that tuning the “reuse” layers by using simple (cid:96) 2 based regularization on previous task parameter (i.e. (cid:107) θ i − θ j (cid:107) 22 , where θ i is the parameter for current task and θ j is the j th task parameter that selected to reuse) is safe enough in terms of forgetting and both strategies can keep the overall accuracy as high as training each task individually (see Fig.

Original: The search are based on the model obtained by training Ima- Next we run a similar experiment on the visual decathlon dataset. All ten tasks are trained in order of ImageNet, CIFAR-100, SVHN, UCF101, Omniglot, GTSR, DPed,
Revised: 3 c).

Original: Flower, Aircraft and Textures. As a baseline we also train a model that shares all parts of the structure.
Revised: Encouraged by the above result, we additionally run experiments by tuning the “reuse” layers without using any regularization .

Original: Both models are trained with simple (cid:96) 2 regularization on model weights between tasks.  As can be seen from ﬁgure 5, the performance from our approach with explicit learning of structure constantly out performs the baseline, suggests a lot less forgetting when transfer between different tasks.
Revised: In other words, we do not add any regularization to the parameters to mitigate forgetting among different tasks. The results is shown in Fig.3 c. As can be observed, here we almost have the same behavior as compared to using the (cid:96) 2 based regularization. This may suggests that the learned architecture actually make sense, and the reused parameter is close to optimal for speciﬁc tasks.

Original: We would like to note
Revised: [Figure]

Original:  that simple (cid:96) 2 based regularization does not prevent forgetting in general in a multi-task setting.
Revised: We would like to note that simple (cid:96)based regularization employed above is not capable of preventing forgetting in general.

Original: This is because using too strong regularization would leads to model parameters that are very close to the original task, which would then prevent the model from learning new tasks.
Revised: This is because using too strong regularization would leads to model parameters that are very close to the original task, which would then prevent the model from learning new tasks.

Original: Smaller regularization helps, however, as the number of tasks increase, the distance between the last model parameter and the very ﬁrst could be signiﬁcant, even though each successive distances are small.
Revised: Smaller regularization helps, however, as the number of tasks increase, the distance between the last model parameter and the very ﬁrst could be significant, even though each successive distances are small.

Original: For baseline model we need to use relatively large regularizations, and thus leads to worse performance of new tasks.
Revised: For baseline model we need to use relatively large regularizations, and thus leads to worse performance of new tasks.

Original: Using smaller regularization in baseline leads to signiﬁcant forgetting.
Revised: Using smaller regularization in baseline leads to signiﬁcant forgetting.

Original: Observing signiﬁcantly less forgetting  may suggests that the model parameters does not move much, or moved on the same level set of the loss surface.
Revised: Observing signiﬁcantly less forgetting in the above experiment may suggests that the model parameters does not move much, or moved on the same level set of the loss surface.

Revised: Next we run a similar experiment on the visual decathlon dataset.

Revised: All ten tasks are trained in order of ImageNet, CIFAR-100, SVHN, UCF101, Omniglot, GTSR, DPed, Flower, Aircraft and Textures.

Revised: As a baseline we also train a model that shares all parts of the structure.

Revised: Both models are trained with similar setting as in previous permuted MNIST experiments, except in our method we choose not to use any regularization this time due to the positive results we got from last experiment.

Original: 4. C OMPARISON WITH O THER M ETHODS
Revised: As can be seen from ﬁgure 6, the performance from our approach with explicit learning of structure constantly out performs the baseline, suggests a lot less forgetting when transfer between different tasks. To see if the ‘reuse’ layers are almost at an optimal position for current task to use, we perform an additional experiment. We calculate the (cid:96) 2 distance between the original parameter and the tuned parameter after each tasks on the VDD dataset. Fig.4 shows (cid:96) 2 distance between the parameters from the very ﬁrst task to each of the tuned parameters in following tasks from the ﬁrst and last layers.

Revised: It is clear that the parameter does not move much from the original location as compared to the baseline, which explains why we observe less forgetting 2 .

Revised: In addition, the moved distance from our methods is more or less the same scale across all layers.

Revised: This may also attribute to the fact that the selection of utilization of parameters (i.e. structure) is explicitly learned, and thus the selected ones are more compatible with current task.

Revised: Therefore, less tuning is required for the new task and hence smaller distance.

Revised: Experiments in this section indicates that learning structure is important.

Revised: With the right structure all the relevant parameters from previous tasks can be used.

Revised: Additionally, since the way to utilize these parameters are learned through structure learning, much less tuning is required for better performance on the new task, and forgetting can thus be minimized.

Revised: 4. C OMPARISON WITH O THER M ETHODS

Revised: In this section, we compare our methods with other recent continuous learning methods – Lee et al.

Revised: (2017b, DEN), Serr`a et al.

Revised: (2018, HAT), Kirkpatrick et al.

Revised: (2017, EWC), Lee et al.

Revised: (2017b, IMM),

Original: In the multi-task continual learning experiments, the 10 tasks was trained in a random sequence except the ﬁrst task was ﬁxed to be
Revised: Rusu et al. (2016, ProgressiveNet), Fernando et al. (2017, PathNet). We compare the performance of various methods on the permuted MNIST dataset with ten different permutations.

Revised: Since our model adds more parameters, for fair comparison we also train other methods with the similar or more number of parameters as compared to our model.

Revised: In particular, since our model tends to add new parameters at the ﬁrst layer.

Revised: For all methods we enlarge the ﬁrst layer hidden size by ten times, so that theoretically they could learn exactly the same structure as our model.

Revised: We ensure that all methods use similar amount of parameters, and the parameter usage of our methods does not exceed any of the other methods that we compare.

Revised: More details regarding this experiment can be found in appendix.

Revised: The results are shown in Fig.

Revised: It is clear that our method (either tuned with or without regularization) performs competitive or better than other methods on this task.

Revised: This result suggests that although theoretically, structure can be learned along with parameter, in practice, the current optimization have a hard time achieving this.

Revised: This in turn indicates the importance of explicit taking structure learning into account when learning tasks continuously.

Revised: Although both DEN and our method dynamically expand the network, our performance is much better, which is attributed to the ability of learning the new structure for different tasks.

Revised: Additionally, our model performs competitive or better as compared to methods that completely avoids forgetting by ﬁxing the learned weights, such as ProgressNet and PathNet, without having such restrictions.

Revised:  R ELATED W ORK

Revised: Continual learning (Thrun & Mitchell, 1995) is a challenging problem, as models have the tendency to forget previously learned knowledge when learning on new information (Thrun & Mitchell, 1995; McCloskey & Cohen, 1989).

Revised: This is referred as catastrophic forgetting problem in the literature.

Revised: Early attempts to alleviate catastrophic forgetting often consists of memory system that store previous data and replay the sampled old examples with the new data (Robins, 1995), and similar approaches are still used today (Rebufﬁ et al., 2017b;

Revised: Li et al., 2018; Lopez-Paz et al., 2017).

Revised: Another for all layers.

Revised: In general, the higher up the layer the more the parameter moves for the baseline, whereas for our method the moved distance is typically very small

Revised: [Figure]

Revised: ID 95.

Revised: [Figure]

Revised: IMM

Revised: [Figure]

Revised: mode

Revised: PN

Revised: PG

Revised: HAT

Revised: DEN

Revised: [Figure]

Revised: [Figure]

Revised: Ours class of common method for mitigating catastrophic forgetting is through regularization which imposes constraints on the

Revised: update of neural weights. Kirkpatrick et al. (2017) proposed elastic weight

Revised: consolidation (EWC), which tries to minimize the change of weights that are important to previous tasks through the use of a quadratic constraint.

Revised: Zenke et al.

Revised: (2017) proposed to alleviate catastrophic forgetting by allowing individual synapse to estimate their importance for solving a learned task, then penalize the change on the important weights.

Revised: Serr ` a et al.

Revised: (2018) proposed method that use attention mechanism to preserve previous’ tasks performance.

Revised: One could also completely avoid forgetting by preventing changes to previous task weights (see for example Rusu et al., 2016; Mallya & Lazebnik, 2018; Fernando et al., 2017).

Revised: Another class of methods for continuous learning is allowing the model to expand.

Revised: Dynamically expandable networks (Lee et al., 2017a) select whether to expand or duplicate based on certain criteria on the new task.

Revised: However, the model on the new task is forced to use the old structure from previous tasks.

Revised: Similarly for progressive networks (Rusu et al., 2016).

Revised: Our framework is more ﬂexible, as it allows model to choose whether to use previous structures.

Revised: Pathnet (Fernando et al., 2017) select paths between predeﬁned modules, and tuning is allowed only when an unused module is selected.

Revised: Our method has more granularity, and does not have any restriction on tuning parameters from previous tasks.

Revised: Our work also relates to neural architecture search (Stanley & Miikkulainen, 2002;

Revised: Zoph & Le, 2016; Baker et al., 2017; Liu et al., 2018), as we employ search methods for the implementation of structure optimization.

Revised: In particular, DARTS (Liu et al., 2018), where a continuous relaxation for architecture search is proposed.

Revised: This allowed more efﬁcient structure optimization, and hence employed in our work.

Revised:  C ONCLUSION

Revised: In this work, we propose to explicit take into account structure optimization in a continuous learning setting.

Revised: Empirically, we found that explicit learning of structure does lead to learning of sensible structures for each of the tasks.

Revised: In particular, lower level features are shared more in case the input representations are more similar, and higher level representations tends to share when the high level semantic has commonality.

Revised: Additionally, the catastrophic forgetting problem gets alleviated with structure optimization.

Revised: We achieved reasonable performance on all tasks, and simultaneously minimized forgetting by only use simple (cid:96) 2 based regularization.

Revised: Further, we show that forgetting is small even without using any regularization on the selected reuse parameters.

Revised: This suggests that structure learning plays a signiﬁcant role in the catastrophic forgetting problem.

Revised: Moreover, we achieved competitive or better performance as compared to all other methods on the permuted

Revised: MNIST dataset.

Revised: Appendices A A DDITIONAL E XPERIMENTAL D ETAILS FOR PERMUTED MNIST

Revised: For all MNIST experiment, we use fully connected layer with three hidden layer, each withhidden units, and one shared output layer for our method.

Revised: For all other methods except pathnet and progressive net we used 3000 units in the ﬁrst layer and 300 for the rest.

Revised: For pathnet, each module in the ﬁrst layer has 300 units, and the result layers has 30 units.

Revised: We use 16 modules per layer, and 5 layers for pathnet, and restrict each mutation to only use 3 modules besides the output layer.

Revised: For progressive net, the ﬁrst layer has 300 units for each task, and the rest layers each hasunits.

Revised: Therefore, all competitive methods are having more or the same number of parameters as our methods.

Revised: For DEN we use the ofﬁcial implementation at https://github.com/jaehong-yoon93/

Revised: DEN , and we used Serr`a et al.

Revised: (2018) implementation of HAT, EWC, and IMM at https:// github.com/joansj/hat .

Revised: We used our own implemention for for Progressive Network and

Original: ImageNet.
Revised: PathNet.

Revised: All methods are trained using the same permutations and same subset of training data.

Revised: B A DDITIONAL E XPERIMENTS ON V ISUAL D ECATHLON D ATASET

Revised: In the multi-task continual learning experiments, the 10 tasks was trained in a random sequence except the ﬁrst task was ﬁxed to be ImageNet.

Original: This is just for fair comparison with other works such as Rebufﬁ et al.
Revised: This is just for fair comparison with other works such as Rebufﬁ et al.

Original: (2017a)
Revised: (2017a)

Original: and Mallya & Lazebnik (2018), they are all using a light weight module to adapt ImageNet pretrained model to other of the 9 tasks.
Revised: and Mallya & Lazebnik (2018), they are all using a light weight module to adapt ImageNet pretrained model to other of the 9 tasks.

Original: In real case, the tasks can come in any order, thus our framework would be much more ﬂexible.
Revised: In real case, the tasks can come in any order, thus our framework would be much more ﬂexible.

Original: As the tasks are trained in sequence, a super model is maintained that all the newly created weights and task-speciﬁc layers are stored.
Revised: As the tasks are trained in sequence, a super model is maintained that all the newly created weights and task-speciﬁc layers are stored.

Original: In this ResNet-26 model, all the Batch Normalization (BN) layers are treated as task-speciﬁc, which means each task has its own sets of BNs.
Revised: In this ResNet-26 model, all the Batch Normalization (BN) layers are treated as task-speciﬁc, which means each task has its own sets of BNs.

Original: Here, we ﬁxed the weight during retraining when ”reuse” is selected in the search phase.
Revised: Here, we ﬁxed the weight during retraining when ”reuse” is selected in the search phase.

Original: This means that the results of previous tasks would not be affected, i.e. no forgetting.
Revised: This means that the results of previous tasks would not be affected, i.e. no forgetting.

Original: We leave the evaluation of forgetting in the context of VDD dataset as future work.
Revised: We leave the evaluation of forgetting in the context of VDD dataset as future work.

Original: In Table 1, we compare the results using our approach with other baselines. ”
Revised: In Table 1, we compare the results using our approach with other baselines. ”

Original: Individual” means that each task is trained individually and weights are initialized randomly. ”
Revised: Individual” means that each task is trained individually and weights are initialized randomly. ”

Original: Classiﬁer” means that only the last layer classiﬁer could be tuned while the former 25 layers are transfer from ImageNet pretrained model and kept ﬁxed during training.
Revised: Classiﬁer” means that only the last layer classiﬁer could be tuned while the former 25 layers are transfer from ImageNet pretrained model and kept ﬁxed during training.

Original: In this case, each task only adds a task-speciﬁc classiﬁer and BNs, thus the overall model size is small. ”
Revised: In this case, each task only adds a task-speciﬁc classiﬁer and BNs, thus the overall model size is small. ”

Original: Adapter” add a 1x1 conv layer besides each 3x3 conv layer, and the outputs will be added before proceed to the next layer.
Revised: Adapter” add a 1x1 conv layer besides each 3x3 conv layer, and the outputs will be added before proceed to the next layer.

Original: Due to the lightweight 1x1 conv layer, each task will add approximately 1/9 of the whole model size.
Revised: Due to the lightweight 1x1 conv layer, each task will add approximately 1/9 of the whole model size.

Original: As shown in table 1, the results achieved by our framework is better than other baselines and the total model size is similar to ”Adapter” case.
Revised: As shown in table 1, the results achieved by our framework is better than other baselines and the total model size is similar to ”Adapter” case.

Original: We can see that our approach gives best results in ﬁve out of nine tasks.
Revised: We can see that our approach gives best results in ﬁve out of nine tasks.

Original: Especially in task with small data size, e.g. VGG-Flowers and Aircraft, our method outperforms other baselines by a large margin.
Revised: Especially in task with small data size, e.g. VGG-Flowers and Aircraft, our method outperforms other baselines by a large margin.

Original: Due to each choice has different parameter cost, we add a parameter loss function to L val to penalize the choices that cost additional parameters.
Revised: Due to each choice has different parameter cost, we add a parameter loss function to L val to penalize the choices that cost additional parameters.

Original: And the value of the loss function is proportional to the product of the additional parameter size and its corresponding weight value α lc .
Revised: And the value of the loss function is proportional to the product of the additional parameter size and its corresponding weight value α lc .

Original: In table 2, we test it with three different scaling factor β of the parameter loss.
Revised: In table 2, we test it with three different scaling factor β of the parameter loss.

Original: We found that the scaling factor β can control the additional parameter size for each task.
Revised: We found that the scaling factor β can control the additional parameter size for each task.

Original: And we ﬁnd that β
Revised: And we ﬁnd that β

Original: = 0 .
Revised: = 0 .

Original: 1 gives the best average accuracy and can control the total model size approximate 2 .
Revised: 1 gives the best average accuracy and can control the total model size approximate 2 .

Original: 3 × compared with the original model size.
Revised: 3 × compared with the original model size.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: R ELATED W ORK

Original: Continual learning (Thrun & Mitchell, 1995) is a challenging problem, as models have the tendency to forget previously learned knowledge when learning on new information (Thrun & Mitchell, 1995; McCloskey & Cohen, 1989).

Original: This is referred as catastrophic forgetting problem in the literature.

Original: Early attempts to alleviate catastrophic forgetting often consists of memory system that store previous data and replay the sampled old examples with the new data (Robins, 1995), and similar approaches are still used today (Rebufﬁ et al., 2017b;

Original: Li et al., 2018; Lopez-Paz et al., 2017).

Original: Another class of common method for mitigating catastrophic forgetting is through regularization which imposes constraints on the update of neural weights.

Original: Kirkpatrick et al.

Original: (2017) proposed elastic weight consolidation (EWC), which tries to minimize the change of weights that are important to previous tasks through the use of a quadratic constraint.

Original: Zenke et al.

Original: (2017) proposed to alleviate catastrophic forgetting by allowing individual synapse to estimate their importance for solving a learned task, then penalize the change on the important weights.

Original: One could also completely avoid forgetting by preventing changes to previous task weights (see for example Rusu et al., 2016; Mallya & Lazebnik, 2018; Fernando et al., 2017).

Original: Another class of methods for continuous learning is allowing the model to expand.

Original: Dynamically expandable networks (Lee et al., 2017) select whether to expand or duplicate based on certain criteria on the new task.

Original: However, the model on the new task is forced to use the old structure from previous tasks.

Original: Our framework is more ﬂexible, as it allows model to choose whether to use previous structures.

Original: Our work also relates to neural architecture search (Stanley & Miikkulainen, 2002;

Original: Zoph & Le, 2016; Baker et al., 2017; Liu et al., 2018), as we employ search methods for the implementation of structure optimization.

Original: In particular, DARTS (Liu et al., 2018), where a continuous relaxation for architecture search is proposed.

Original: This allowed more efﬁcient structure optimization, and hence employed in our work.

Original: C ONCLUSION

Original: In this work we propose to explicit take into account structure optimization in a continuous learning setting.

Original: Empirically, we found that explicit learning of structure does lead to learning of sensible structures for each of the tasks.

Original: In particular, lower level features are shared more in case the input representations are more similar, and higher level representations tends to share when the high level semantic has commonality.

Original: Additionally, the catastrophic forgetting problem gets alleviated with structure optimization.

Original: We achieved reasonable performance on all tasks and simultaneously minimized forgetting by only use simple (cid:96) 2 based regularization.

Original: This suggests that structure learning plays a signiﬁcant role in the catastrophic forgetting problem.

Original: Moreover, we achieved competitive or better performance as compared to all baselines on the visual decathlon dataset