Original: Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer
Revised: Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer

Original: Abstract
Revised: Abstract

Original: Effective lifelong learning across diverse tasks requires diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forgetting.
Revised: Effective lifelong learning across diverse tasks requires diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forgetting.

Original: In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must be driven by the relationships among tasks.
Revised: In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must be driven by the relationships among tasks.

Original: We ﬁrst show that the lifelong learning performance of several current deep learning architectures can be signiﬁcantly improved by transfer at the appropriate layers.
Revised: We ﬁrst show that the lifelong learning performance of several current deep learning architectures can be signiﬁcantly improved by transfer at the appropriate layers.

Original: We then develop an expectation-maximization (EM) method to automatically select the appropriate transfer conﬁguration and optimize the task network weights.
Revised: We then develop an expectation-maximization (EM) method to automatically select the appropriate transfer conﬁguration and optimize the task network weights.

Original: This
Revised: This

Original: EM-based selective transfer is highly effective, as demonstrated on three algorithms in several lifelong object classiﬁcation scenarios.
Revised: EM-based selective transfer is highly effective, as demonstrated on three algorithms in several lifelong object classiﬁcation scenarios.

Original: 1. Introduction
Revised: 1. Introduction

Original: Transfer at different layers within a deep network corresponds to sharing knowledge between tasks at different levels of abstraction.
Revised: Transfer at different layers within a deep network corresponds to sharing knowledge between tasks at different levels of abstraction.

Original: In multi-task scenarios that involve diverse tasks, reusing low-layer representations may be appropriate for tasks that share feature-based similarities, while sharing high-level representations may be more appropriate for tasks that share more abstract similarities.
Revised: In multi-task scenarios that involve diverse tasks, reusing low-layer representations may be appropriate for tasks that share feature-based similarities, while sharing high-level representations may be more appropriate for tasks that share more abstract similarities.

Original: Selecting the appropriate granularity of knowledge to transfer is an important architectural consideration for deep networks that support multiple tasks.
Revised: Selecting the appropriate granularity of knowledge to transfer is an important architectural consideration for deep networks that support multiple tasks.

Original: In scenarios where tasks share substantial similarities, many multi-task methods have found success using a universal conﬁguration of the knowledge sharing (Caruana, 1993;
Revised: In scenarios where tasks share substantial similarities many multi-task methods have found success using a universal conﬁguration of the knowledge sharing (Caruana, 1993;

Original: Yang and Hospedales, 2017; Lee et al., 2019; Liu et al., 2019; Bulat et al., 2020), such as sharing the lower layersAnonymous Region, Anonymous
Revised: Yang and Hospedales, 2017; Lee et al., 2019; Liu et al., 2019; Bulat et al., 2020), such as sharing the lower layers

Original: Preliminary work.

Original: Under review by the International Conference on

Original: ICML).

Original: Do not distribute.

Original:  of a deep network with upper-level task-speciﬁc heads.
Revised: Corre- of a deep network with upper-level task-speciﬁc heads.

Original: As tasks become increasingly diverse, the appropriate granularity for transfer may vary between tasks based on their relationships, necessitating more selective transfer.
Revised: As tasks become increasingly diverse, the appropriate granularity for transfer may vary between tasks based on their relationships, necessitating more selective transfer.

Original: Prior work in selective sharing for deep networks has typically either (1) branched the network into a tree structure (Lu et al., 2017; Yoon et al., 2018; Vandenhende et al., 2019;
Revised: Prior work in selective sharing for deep networks has typically either (1) branched the network into a tree structure (Lu et al., 2017; Yoon et al., 2018; Vandenhende et al., 2019;

Original: He et al., 2018), which emphasizes the sharing of lower layers or (2) introduced new learning modules between task models (Yang and Hospedales, 2017; Xiao et al., 2018; Cao et al., 2018; Rusu et al., 2016) which increases the complexity of training.
Revised: He et al., 2018), which emphasizes the sharing of lower layers or (2) introduced new learning modules between task models (Yang and Hospedales, 2017; Xiao et al., 2018; Cao et al., 2018; Rusu et al., 2016) which increases the complexity of training.

Original: The transfer conﬁguration could then be optimized in batch settings to maximize performance across the tasks.
Revised: The transfer conﬁguration could then be optimized in batch settings to maximize performance across the tasks.

Original: However, the problem of selective transfer is further compounded in continual or lifelong learning settings, in which tasks are presented sequentially.
Revised: However, the problem of selective transfer is further compounded in continual or lifelong learning settings, in which tasks are presented sequentially.

Original: The optimal transfer conﬁguration may vary between tasks or over time.
Revised: The optimal transfer conﬁguration may vary between tasks or over time.

Original: To verify this premise and motivate our work, we conducted a simple experiment: we took a multi-task CNN with shared layers and a lifelong learning CNN that uses factorized transfer (DF-CNN (Lee et al., 2019)) and varied the set of CNN layers that employed transfer (with task-speciﬁc fully connected layers at the top).
Revised: To verify this premise and motivate our work, we conducted a simple experiment: we took a multi-task CNN with shared layers and a lifelong learning CNN that uses factorized transfer (DF-CNN (Lee et al., 2019)) and varied the set of CNN layers that employed transfer (with task-speciﬁc fully connected layers at the top).

Original: Using two data sets, we considered transferring at all CNN layers, transfer at the top- k CNN layers, transfer at the bottom- k CNN layers, and alternating transfer/no-transfer CNN layers.
Revised: Using two data sets, we considered transferring at all CNN layers, transfer at the top- k CNN layers, transfer at the bottom- k CNN layers, and alternating transfer/no-transfer CNN layers.

Original: The results are shown in
Revised: The results are shown in

Original: Figure 1, with details given in Section 2.
Revised: Figure 1, with details given in Section 2.

Original: Clearly, we see that the optimal transfer conﬁguration varies between task relationships and transfer mechanisms.
Revised: Clearly, we see that the optimal transfer conﬁguration varies between task relationships and transfer mechanisms.

Original: Restricting the transfer layers signiﬁcantly improves performance over the na¨ıve approach of transferring at all layers, with the alternating conﬁguration performing extremely well in both cases.
Revised: Restricting the transfer layers signiﬁcantly improves performance over the na¨ıve approach of transferring at all layers, with the alternating conﬁguration performing extremely well in both cases.

Original: To enable a more ﬂexible version of selective transfer, we investigate the use of architecture search to dynamically adjust the transfer conﬁguration between tasks and over time.
Revised: To enable a more ﬂexible version of selective transfer, we investigate the use of architecture search to dynamically adjust the transfer conﬁguration between tasks and over time.

Original: We use expectation-maximization (EM) to learn both the parameters of the task models and the layers to transfer within the deep net.
Revised: We use expectation-maximization (EM) to learn both the parameters of the task models and the layers to transfer within the deep net.

Original: This approach, Lifelong Architecture Search via EM (LASEM), enables deep networks to transfer different sets of layers for each task, allowing more ﬂexibility over branching-based conﬁgurations for selective transfer.
Revised: This approach, Lifelong Architecture Search via EM (LASEM), enables deep networks to transfer different sets of layers for each task, allowing more ﬂexibility over branching-based conﬁgurations for selective transfer.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: k P e r - t a s k A cc u r a c y

Original: [Figure]
Revised: [Table]

Original: D F - C NN a ll D F - C NN t o p
Revised: [Table]

Original: D F - C NN t o p D F - C NN t o p
Revised: [Table] [Table] [Table]

Original: D F - C NN b o tt o m D F - C NN b o tt o m
Revised: [Table] [Table] [Table]

Original: D F - C NN b o tt o m D F - C NN a l t e r .
Revised: r .

Original: M e a n P e a k P e r - t
Revised: M e a n P e a k P e r - t

Original: a s k A cc u r a c y 
Revised: a s k A cc u r a c y (

Original: [Figure]
Revised: [Figure]

Original: in a lifelong learning setting with 95% conﬁdence interval.
Revised: Ofﬁce

Original: This empirically shows that the optimal transfer con- varies, and choosing the correct conﬁguration is superior to transfer at all layers.
Revised: [Figure] Home

Original: It also introduces little additional computation in comparison to exhaustive search over all transfer conﬁgurations or training selective transfer modules between task networks.
Revised: It also introduces little additional computation in comparison to exhaustive search over all transfer conﬁgurations or training selective transfer modules between task networks.

Original: To demonstrate its effectiveness, we applied LASEM to three architectures that support knowledge transfer between tasks in several lifelong learning scenarios and compared it against other lifelong learning and architecture search methods.
Revised: To demonstrate its effectiveness, we applied LASEM to three architectures that support knowledge transfer between tasks in several lifelong learning scenarios and compared it against other lifelong learning and architecture search methods.

Original: 2. The effect of different transfer conﬁgurations in lifelong learning
Revised: 2. The Effect of Different Transfer Conﬁgurations in Lifelong Learning

Original: This section further describes the experiments mentioned in the introduction as motivation for our proposed LASEM method.
Revised: This section further describes the experiments mentioned in the introduction as motivation for our proposed LASEM method.

Original: The hypothesis of our work is that lifelong deep learning can beneﬁt from using a more ﬂexible transfer mechanism that selectively chooses the transfer architecture for each task.
Revised: The hypothesis of our work is that lifelong deep learning can beneﬁt from using a more ﬂexible transfer mechanism that selectively chooses the transfer architecture for each task.

Original: This would permit it to dynamically select, for each task model, which layers to transfer and which to keep as task-speciﬁc (enabling it to customize transferred knowledge to an individual task).
Revised: This would permit it to dynamically select, for each task model, which layers to transfer and which to keep as task-speciﬁc (enabling it to customize transferred knowledge to an individual task).

Original: To determine the effect of different transfer conﬁgurations, we conducted a set of initial experiments using two established methods:
Revised: To determine the effect of different transfer conﬁgurations, we conducted a set of initial experiments using two established methods:

Original: Multi-Task CNN with hard parameter sharing (HPS):
Revised: Multi-Task CNN with hard parameter sharing (HPS):

Original: This approach shares the hidden CNN layers between all tasks, and maintains task-speciﬁc fully connected output layers.
Revised: This approach shares the hidden CNN layers between all tasks, and maintains task-speciﬁc fully connected output layers.

Original: It is one of the most common methods for multi-task learning of neural networks (Caruana, 1993; 1997), and is widely used.
Revised: It is one of the most common methods for multi-task learning of neural networks (Caruana, 1993; 1997), and is widely used.

Original: Deconvolutional factorized CNN (DF-CNN):
Revised: Deconvolutional factorized CNN (DF-CNN):

Original: The DFCNN (Lee et al., 2019) adapts CNNs to a continual learning setting by sharing layer-wise knowledge across tasks.
Revised: The DFCNN (Lee et al., 2019) adapts CNNs to a continual learning setting by sharing layer-wise knowledge across tasks.

Original: The convolutional ﬁlters of each task model are dynamically generated from a task- independent layer- dependent shared tensor through a series of task-speciﬁc mapping.
Revised: The convolutional ﬁlters of each task model are dynamically generated from a task- independent layer- dependent shared tensor through a series of task-speciﬁc mapping.

Original: When training the task models consecutively, gradients ﬂow through to update the shared tensors and the task-speciﬁc parameters, so this transfer architecture enables the DF-CNN to learn and compress knowledge universal among the observed tasks into the shared tensors.
Revised: When training the task models consecutively, gradients ﬂow through to update the shared tensors and the task-speciﬁc parameters, so this transfer architecture enables the DF-CNN to learn and compress knowledge universal among the observed tasks into the shared tensors.

Original: Both these methods utilize a set of transfer-based CNN layers and non-transfer task-speciﬁc layers.
Revised: Both these methods utilize a set of transfer-based CNN layers and non-transfer task-speciﬁc layers.

Original: For a network with d CNN layers, there are 2 d potential transfer conﬁgurations.
Revised: For a network with d CNN layers, there are 2 d potential transfer conﬁgurations.

Original: To explore the effect of transfer at different layers, we varied the transfer conﬁguration among several options:
Revised: To explore the effect of transfer at different layers, we varied the transfer conﬁguration among several options:

Original: • All : Transfer at all CNN layers.
Revised: • All : Transfer at all CNN layers.

Original: Note that the original DF-CNN used this conﬁguration.
Revised: Note that the original DF-CNN used this conﬁguration.

Original: • Top k : Transfer across task models occurs only at the k highest CNN layers, with all others remaining taskspeciﬁc.
Revised: • Top k : Transfer across task models occurs only at the k highest CNN layers, with all others remaining taskspeciﬁc.

Original: We would expect this transfer conﬁguration to beneﬁt tasks that share high-level concepts but have low-level feature differences.
Revised: We would expect this transfer conﬁguration to beneﬁt tasks that share high-level concepts but have low-level feature differences.

Original: • Bottom k : Transfer occurs only at the k lowest CNN layers, which is opposite of the Top d − k .
Revised: • Bottom k : Transfer occurs only at the k lowest CNN layers, which is opposite of the Top d − k .

Original: We would expect it to beneﬁt tasks that share perceptual similarities but have high-level differences.
Revised: We would expect it to beneﬁt tasks that share perceptual similarities but have high-level differences.

Original: Alternating :
Revised: Alternating :

Original: This conﬁguration alternates transfer and non-transfer layers, enabling the non-transfer taskspeciﬁc layers to further customize the transferred knowledge to the task.
Revised: This conﬁguration alternates transfer and non-transfer layers, enabling the non-transfer taskspeciﬁc layers to further customize the transferred knowledge to the task.

Original: We evaluate the performance of various transfer conﬁgurations on the CIFAR-100 (Krizhevsky and Hinton, 2009) and
Revised: We evaluate the performance of various transfer conﬁgurations on the CIFAR-100 (Krizhevsky and Hinton, 2009) and

Original: Ofﬁce-Home (Venkateswara et al., 2017) data sets, following the lifelong learning experimental setup used in previous work (Lee et al., 2019).
Revised: Ofﬁce-Home (Venkateswara et al., 2017) data sets, following the lifelong learning experimental setup used in previous work (Lee et al., 2019).

Original: CIFAR-100 involves ten consecutive tasks of ten-way image classiﬁcation, where any object class occurs in only one task.
Revised: CIFAR-100 involves ten consecutive tasks of ten-way image classiﬁcation, where any object class occurs in only one task.

Original: Ofﬁce-Home involves ten tasks of thirteen-way classiﬁcation, separated into two domains: ‘Product’ images and ‘Real World’ images.
Revised: Ofﬁce-Home involves ten tasks of thirteen-way classiﬁcation, separated into two domains: ‘Product’ images and ‘Real World’ images.

Original: The
Revised: The

Original: CNN architectures used for each data set and optimization settings follow prior work (Lee et al., 2019).
Revised: CNN architectures used for each data set and optimization settings follow prior work (Lee et al., 2019).

Original: During training, we measured the peak per-task accuracy on held-out test data, averaging over ﬁve trials.
Revised: During training, we measured the peak per-task accuracy on held-out test data, averaging over ﬁve trials.

Original: Our results, shown in Figure 1, reveal that permitting transfer at all layers does not guarantee the best performance.
Revised: Our results, shown in Figure 1, reveal that permitting transfer at all layers does not guarantee the best performance.

Original: This observation complicates learning on novel tasks, since the best transfer conﬁguration depends both on the algorithm and the task relations in the data set.
Revised: This observation complicates learning on novel tasks, since the best transfer conﬁguration depends both on the algorithm and the task relations in the data set.

Original: Notably, we see that the DF-CNN, which is designed for lifelong learning, can be improved beyond the original version (Lee et al., 2019) by allowing transfer at only some layers.
Revised: Notably, we see that the DF-CNN, which is designed for lifelong learning, can be improved beyond the original version (Lee et al., 2019) by allowing transfer at only some layers.

Original: Furthermore, we can see that the optimal transfer conﬁguration varies between data sets and algorithms.
Revised: Furthermore, we can see that the optimal transfer conﬁguration varies between data sets and algorithms.

Original: For instance on Ofﬁce-Home, sharing lower layers in the HPS multi-task CNN achieves better performance on average, but transferring upper layers works better for the DF-CNN.
Revised: For instance on Ofﬁce-Home, sharing lower layers in the HPS multi-task CNN achieves better performance on average, but transferring upper layers works better for the DF-CNN.

Original: Similarly, the Alternating conﬁguration consistently achieves near the best performance for the DF-CNN, beneﬁting from permitting the non-transfer layers to customize transferred knowledge to the individual task, but it is not consistently as good for HPS.
Revised: Similarly, the Alternating conﬁguration consistently achieves near the best performance for the DF-CNN, beneﬁting from permitting the non-transfer layers to customize transferred knowledge to the individual task, but it is not consistently as good for HPS.

Original: 3. Architecture search for the optimal transfer conﬁguration
Revised: 3. Architecture Search for the Optimal Transfer Conﬁguration

Original: The experiment presented above reveals that the transfer conﬁguration can have a signiﬁcant effect on lifelong learning performance, and that the best transfer conﬁguration varies.
Revised: The experiment presented above reveals that the transfer conﬁguration can have a signiﬁcant effect on lifelong learning performance, and that the best transfer conﬁguration varies.

Original: These observations inspire our work to develop a more ﬂexible mechanism for selective transfer in lifelong learning by viewing the transfer conﬁguration as a new hyper-parameter for each task model.
Revised: These observations inspire our work to develop a more ﬂexible mechanism for selective transfer in lifelong learning by viewing the transfer conﬁguration as a new hyper-parameter for each task model.

Original: The search space grows exponentially as the neural network gets deeper (i.e., 2 d conﬁgurations for d CNN layers) and linearly as the more tasks are learned.
Revised: The search space grows exponentially as the neural network gets deeper (i.e., 2 d conﬁgurations for d CNN layers) and linearly as the more tasks are learned.

Original: Formally, a layer-based transfer conﬁguration for task t can be speciﬁed by a d -dimensional binary vector c t ∈ C
Revised: Formally, a layer-based transfer conﬁguration for task t can be speciﬁed by a d -dimensional binary vector c t ∈ C

Original: = { 0 , 1 } d , where each c t,j is a binary indicator whether or not the j th layer involves transfer.
Revised: = { 0 , 1 } d , where each c t,j is a binary indicator whether or not the j th layer involves transfer.

Original: We can compactly notate c t by a set of its indices containing True entries.
Revised: We can compactly notate c t by a set of its indices containing True entries.

Original: For example, the Alternating conﬁguration described in Section 2 can be denoted by c = [0 , 1 , 0 , 1] = { 2 , 4 } (refer to Figure 2).
Revised: For example, the Alternating conﬁguration described in Section 2 can be denoted by c = [0 , 1 , 0 , 1] = { 2 , 4 } (refer to Figure 2).

Original: Our goal is to optimize the task-speciﬁc transfer conﬁguration while simultaneously optimizing the parameters of the task models and shared knowledge in a lifelong setting.
Revised: Our goal is to optimize the task-speciﬁc transfer conﬁguration while simultaneously optimizing the parameters of the task models and shared knowledge in a lifelong setting.

Original: Treating c t as a latent variable of the model for task t , we employ expectation-maximization (EM) to perform this joint optimization.
Revised: Treating c t as a latent variable of the model for task t , we employ expectation-maximization (EM) to perform this joint optimization.

Original: For each new task, LASEM maintains a set of transfer-based parameters θ ( l ) s and task-speciﬁc parameters θ ( l ) t for each layer l , using the chosen conﬁguration c t to determine which combination of parameters will be used to form the speciﬁc model.
Revised: For each new task, LASEM maintains a set of transfer-based parameters θ ( l ) s and task-speciﬁc parameters θ ( l ) t for each layer l , using the chosen conﬁguration c t to determine which combination of parameters will be used to form the speciﬁc model.

Original: In brief, the E-step estimates the usefulness of the representation that each transfer conﬁguration c i ∈ C can learn from the given data (i.e., the likelihood of data), while the M-step optimizes parameters of the task model and shared knowledge.
Revised: In brief, the E-step estimates the usefulness of the representation that each transfer conﬁguration c i ∈ C can learn from the given data (i.e., the likelihood of data), while the M-step optimizes parameters of the task model and shared knowledge.

Original: We ﬁrst consider how to model the prior π t on possible conﬁgurations of the current task’s c t .
Revised: We ﬁrst consider how to model the prior π t on possible conﬁgurations of the current task’s c t .

Original: Using a simple frequency-based probability estimate with Laplace smoothing represents the prior probability of each conﬁguration as
Revised: Using a simple frequency-based probability estimate with Laplace smoothing represents the prior probability of each conﬁguration as

Original: [Equation]
Revised: [Equation]

Original: where c ( t ) denotes the conﬁguration for task t , and n c i is the number of mini-batches whose most probable conﬁguration is c i .
Revised: where c ( t ) denotes the conﬁguration for task t , and n c i is the number of mini-batches whose most probable conﬁguration is c i .

Original: This estimate considers each transfer conﬁguration solely based on the current task’s data, but alternative priors could instead be used, such as measuring the most frequent transfer conﬁguration historically over all tasks or measuring the most frequent conﬁguration over related tasks (which requires a notion of task similarity, such as via a task descriptor (Isele et al., 2016; Sinapov et al., 2015)).
Revised: This estimate considers each transfer conﬁguration solely based on the current task’s data, but alternative priors could instead be used, such as measuring the most frequent transfer conﬁguration historically over all tasks or measuring the most frequent conﬁguration over related tasks (which requires a notion of task similarity, such as via a task descriptor (Isele et al., 2016; Sinapov et al., 2015)).

Original: In the E-step, the posterior on conﬁgurations is derived by combining the above prior and likelihood, which can be computed from the output of the task network on the current task’s data D new :=
Revised: In the E-step, the posterior on conﬁgurations is derived by combining the above prior and likelihood, which can be computed from the output of the task network on the current task’s data D new :=

Original: [Equation]
Revised: [Equation]

Original: X new ,

Original: [Equation]
Revised: [Equation]

Revised: HPS

Revised: TF DF-CNN

Revised: HPS

Revised: TF DF-CNN

Original: new ) :
Revised: TF DF-CNN 0. M e a n P e a k P e r - t a s k A cc u r a c y

Revised: CIFAR100 4% data

Original: [Equation]
Revised: O [Figure] Home [Figure]

Revised: % data

Revised: STL10 20% data

Original: [Figure]
Revised: [Figure]

Revised: at

Revised: [Figure]

Original: Architecture
Revised: Architecture

Original: Acc.
Revised: Acc.

Original: Rel.
Revised: Rel.

Original: Acc.
Revised: Acc.

Original: CIFAR-100 (10 Tasks)
Revised: CIFAR-100 (10 Tasks)

Original: HPS 39.3 ± 0.
Revised: HPS 39.3 ± 0.

Original: TF 38.4 ± 0.
Revised: TF 38.4 ± 0.

Original: DF-CNN 42.0 ± 0.
Revised: DF-CNN 42.0 ± 0.

Original: Ofﬁce-Home (10 Tasks)
Revised: Ofﬁce-Home (10 Tasks)

Original: HPS 58.4 ± 0.
Revised: HPS 58.4 ± 0.

Original: TF 59.1 ± 1.
Revised: TF 59.1 ± 1.

Original: DF-CNN 59.5 ± 1.
Revised: DF-CNN 59.5 ± 1.

Original: STL-10 (20 Tasks)
Revised: STL-10 (20 Tasks)

Original: TF 71.0 ± 0.
Revised: TF 71.0 ± 0.

Original: [Figure]

Original: [Figure] [Figure]
Revised: DF-CNN 70.7 ± 0.

Original: The M-step improves the log-likelihood viathe estimated probability distribution overthe transfer conﬁgurations.
Revised: M step improves log- via probability distribution over transfer conﬁgurations

Original: Both θ s and θ t are updated by the aggregated gradients of the log-likelihood in cases where the transfer conﬁgurations match the corresponding parameter.
Revised: Both θ s and θ t are updated by the aggregated gradients of the log-likelihood in cases where the transfer conﬁgurations match the corresponding parameter.

Original: To combine the gradients of a speciﬁc parameter tensor over multiple possible conﬁgurations, we take the sum of the corresponding gradients weighted by the above posterior estimate.θ
Revised: To combine the gradients of a speciﬁc parameter tensor over multiple possible conﬁgurations, we take the sum of the corresponding gradients weighted by the above posterior estimate.

Original: [Equation] s ← θ ( l [Equation] + λ (cid:80)
Revised: θ ( l ) (cid:48) s ← θ ( l ) s + λ (cid:88)

Original: i 
Revised: i : c i,l = P ( c i

Original: [Equation]
Revised: [Equation]

Original:  ) θ ( l ) (cid:48) t ← θ ( l ) t + λ (cid:80) i :
Revised: D new ) ∇ log L ( D new | c i ) θ ( l ) (cid:48) t ← θ ( l ) t + λ (cid:88)

Original:  c i,l =0 P ( c i |D new ) ∇ log L ( D new | c i ) (3) for l ∈ { 1 , · · · , d } .
Revised: i : c i,l = P ( c i |D new ) ∇ log L ( D new | c i ) (3) for l ∈ { 1 , · · · , d } .

Original: Here, L ( D new | c i ) is the likelihood of data given
Revised: Here, L ( D new | c i ) is the likelihood of data given

Original: conﬁguration c
Revised: conﬁguration c

Original: i : P ( y new | X new , c i ) .
Revised: i : P ( y new | X new , c i ) .

Original: The main difference in the update rules in Equation 3 is the condition for the index of the summation.
Revised: The main difference in the update rules in Equation 3 is the condition for the index of the summation.

Original: Our LASEM follows lifelong learning framework in which the learner encounters tasks sequentially.
Revised: Our LASEM follows lifelong learning framework in which the learner encounters tasks sequentially.

Original: The parameters of the transfer-based components are initialized once at the beginning of the learning process, while the parameters of the task-speciﬁc components and prior probability of conﬁgurations are initialized every new task.
Revised: The parameters of the transfer-based components are initialized once at the beginning of the learning process, while the parameters of the task-speciﬁc components and prior probability of conﬁgurations are initialized every new task.

Original: LASEM iterates E-
Revised: LASEM iterates E-

Original: and
Revised: and

Original: M-steps on data of the current task until task switches, and the best transfer conﬁguration for the task is determined by the learned posterior probability.
Revised: M-steps on data of the current task until task switches, and the best transfer conﬁguration for the task is determined by the learned posterior probability.

Original: To reduce the complexity of computation, LASEM uses one set of transfer-based and task-speciﬁc parameters ( θ s and θ t ) for all transfer conﬁgurations, rather than maintaining distinct sets of parameters for each conﬁguration.
Revised: To reduce the complexity of computation, LASEM uses one set of transfer-based and task-speciﬁc parameters ( θ s and θ t ) for all transfer conﬁgurations, rather than maintaining distinct sets of parameters for each conﬁguration.

Original: 4. Experiments
Revised: 4. Experiments

Original: We evaluated Lifelong Architecture Search via EM (LASEM) following the same experimental protocol for lifelong learning as used in Section 2.
Revised: We evaluated Lifelong Architecture Search via EM (LASEM) following the same experimental protocol for lifelong learning as used in Section 2.

Original: In addition to using
Revised: In addition to using

Original: CIFAR100 and Ofﬁce-Home, we introduce a lifelong learning version of the STL-10 data set (Coates et al., 2011).
Revised: CIFAR100 and Ofﬁce-Home, we introduce a lifelong learning version of the STL-10 data set (Coates et al., 2011).

Original: STL10 has 5,000 training and 8,000 test images divided evenly among 10 classes, with higher resolution than CIFAR-100.
Revised: STL10 has 5,000 training and 8,000 test images divided evenly among 10 classes, with higher resolution than CIFAR-100.

Original: We constructed 20 tasks of three-way classiﬁcation using 20% and 5% of the given training data for training and validation, respectively.
Revised: We constructed 20 tasks of three-way classiﬁcation using 20% and 5% of the given training data for training and validation, respectively.

Original: To increase the task variations, for each task we randomly chose three images classes, applied
Revised: To increase the task variations, for each task we randomly chose three images classes, applied

Original: Gaussian noise to the images with a random mean and variance, and randomly permuted the channels.
Revised: Gaussian noise to the images with a random mean and variance, and randomly permuted the channels.

Original: All results were averaged over ﬁve trials with random task orders.
Revised: All results were averaged over ﬁve trials with random task orders.

Original: Details of these three experiments as well as architectures of neural networks can be found in Appendix A.
Revised: Details of these three experiments as well as architectures of neural networks can be found in Appendix A.

Original: 4.1. Performance of LASEM
Revised: 4.1. Performance of LASEM

Original: We applied LASEM to three lifelong learning architectures: a multi-task CNN with hard-parameter sharing (HPS) (Caruana, 1997), Tensor Factorization (TF)
Revised: We applied LASEM to three lifelong learning architectures: a multi-task CNN with hard-parameter sharing (HPS) (Caruana, 1997), Tensor Factorization (TF)

Original: (Yang and
Revised: (Yang and

Original: Hospedales, 2017; Bulat et al., 2020) and the Deconvolutional Factorized CNN (DF-CNN)
Revised: Hospedales, 2017; Bulat et al., 2020) and the Deconvolutional Factorized CNN (DF-CNN)

Original: (Lee et al., 2019).
Revised: (Lee et al., 2019).

Original: HPS interconnects CNNs in tree structures, with task models explicitly using the same parameters of all shared layers.
Revised: HPS interconnects CNNs in tree structures, with task models explicitly using the same parameters of all shared layers.

Original: In contrast, the TF and DF-CNN task models explicitly share only a fraction of tensors, and parameters of each task model are generated via transfer.Selective
Revised: In contrast, the TF and DF-CNN task models explicitly share only a fraction of tensors, and parameters of each task model are generated via transfer.

Original: [Table]
Revised: [Table]

Original: Figure 3 compares the performance of the task-speciﬁc transfer conﬁgurations discovered by LASEM (red) to using a single static transfer conﬁguration(black boxes).
Revised: Figure 3 compares the performance of the task-speciﬁc transfer conﬁgurations discovered by LASEM (red) to using a single static transfer conﬁguration(black boxes).

Original: These black boxes depict the performance range of the methods using various transfer conﬁgurations (i.e., All, Top k , Bottom k , Alternating) for all task models, with All shown in blue.
Revised: These black boxes depict the performance range of the methods using various transfer conﬁgurations (i.e., All, Top k , Bottom k , Alternating) for all task models, with All shown in blue.

Original: To estimate this range, we tested eight (50%) and(25%) of the possible static conﬁgurations for the four-CNNlayer (CIFAR-100 and Ofﬁce-Home) and six-CNN-layer (STL-10) task models, respectively.
Revised: To estimate this range, we tested eight (50%) and(25%) of the possible static conﬁgurations for the four-CNNlayer (CIFAR-100 and Ofﬁce-Home) and six-CNN-layer (STL-10) task models, respectively.

Original: We can see that LASEM chose transfer conﬁgurations that perform toward the top of each range, especially on the
Revised: We can see that LASEM chose transfer conﬁgurations that perform toward the top of each range, especially on the

Original: DF-CNN designed for lifelong learning.
Revised: DF-CNN designed for lifelong learning.

Original: LASEM clearly outperforms transfer at all layers.
Revised: LASEM clearly outperforms transfer at all layers.

Original: Automatically selecting the transfer conﬁguration becomes even more beneﬁcial for methods that have a wide range of performances for different static conﬁgurations.
Revised: Automatically selecting the transfer conﬁguration becomes even more beneﬁcial for methods that have a wide range of performances for different static conﬁgurations.

Original: Moreover, LASEM imposes little additional cost of computation in order to determine the transfer conﬁguration.
Revised: Moreover, LASEM imposes little additional cost of computation in order to determine the transfer conﬁguration.

Original: In timing experiments, we found that, compared to training with a pre-determined static transfer conﬁguration, LASEM requires only 30% additional wallclock time to search over 16 conﬁgurations of a network with four CNN layers, and only double the time to search over 64 conﬁgurations of a network with six CNN layers.
Revised: In timing experiments, we found that, compared to training with a pre-determined static transfer conﬁguration, LASEM requires only 30% additional wall-clock time to search over 16 conﬁgurations of a network with four CNN layers, and only double the time to search over 64 conﬁgurations of a network with six CNN layers.

Original: Brute force search over conﬁgurations requires 15 × and 63 × additional time per task, respectively, over the base learner.
Revised: Brute force search over conﬁgurations requires 15 × and 63 × additional time per task, respectively, over the base learner.

Revised: Additional analysis, including results on catastrophic forgetting, can be found in Appendix B.

Original: 4.2. Comparison to other selective transfer algorithms
Revised: 4.2. Comparison to other selective transfer algorithms

Original: We further compared the performance of LASEM against other methods that employ some notion of selective transfer, including the Dynamically Expandable Network (DEN) (Yoon et al., 2018), the Progressive Neural Net (ProgNN) (Rusu et al., 2016) and Differentiable Architecture Search (DARTS) (Liu et al., 2018), on Ofﬁce-Home.
Revised: We further compared the performance of LASEM against other methods that employ some notion of selective transfer, including the Dynamically Expandable Network (DEN) (Yoon et al., 2018), the Progressive Neural Net (ProgNN) (Rusu et al., 2016) and Differentiable Architecture Search (DARTS) (Liu et al., 2018), on Ofﬁce-Home.

Original: DEN is a lifelong learning architecture that extends tree structure sharing of HPS by expanding, splitting, and selectively retraining the network to introduce both shared and task-speciﬁc parameters in each layer if required.
Revised: DEN is a lifelong learning architecture that extends tree structure sharing of HPS by expanding, splitting, and selectively retraining the network to introduce both shared and task-speciﬁc parameters in each layer if required.

Original: ProgNN learns additional layer-wise lateral connections from earlier task models to the current task model, which support one directional transfer to avoid negative knowledge transfer (known as catastrophic forgetting).
Revised: ProgNN learns additional layer-wise lateral connections from earlier task models to the current task model, which support one directional transfer to avoid negative knowledge transfer (known as catastrophic forgetting).

Original: Both DEN and ProgNN can support complex transfer conﬁgurations due to their lack of constraints, such as no assumption of a tree-structured conﬁguration.
Revised: Both DEN and ProgNN can support complex transfer conﬁgurations due to their lack of constraints, such as no assumption of a tree-structured conﬁguration.

Original: For example, a ProgNN with all zero-weighted lateral connections for a level creates a task-speciﬁc layer, and zero-weighted current task model connections creates a transfer-based layer.
Revised: For example, a ProgNN with all zero-weighted lateral connections for a level creates a task-speciﬁc layer, and zero-weighted current task model connections creates a transfer-based layer.

Original: DARTS is the gradient-based framework for neural architecture search, determining both the most suitable operation of each layer and the best architecture of stacking these layers.
Revised: DARTS is the gradient-based framework for neural architecture search, determining both the most suitable operation of each layer and the best architecture of stacking these layers.

Original: To enable the selection of architecture learnable, DARTS introduces a soft selection of the operators, making each layer a weighted sum of operations.
Revised: To enable the selection of architecture learnable, DARTS introduces a soft selection of the operators, making each layer a weighted sum of operations.

Original: Table 1 summarizes the performance of these methods and our approach.
Revised: Table 1 summarizes the performance of these methods and our approach.

Original: ProgNN and LASEM DF-CNN are statistically indistinguishable and perform better than the other methods.
Revised: ProgNN and LASEM DF-CNN are statistically indistinguishable and perform better than the other methods.

Original: LASEM DF-CNN is ∼ 14% faster than ProgNN for learning ten tasks.
Revised: LASEM DF-CNN is ∼ 14% faster than ProgNN for learning ten tasks.

Original: This gap could become larger as more tasks are learned because time complexity of ProgNN is proportional to the square of the number of tasks while the complexity of DF-CNN is linear.
Revised: This gap could become larger as more tasks are learned because time complexity of ProgNN is proportional to the square of the number of tasks while the complexity of DF-CNN is linear.

Original: DEN and DARTS have better training times, but fail to perform as well.
Revised: DEN and DARTS have better training times, but fail to perform as well.

Original: Note that
Revised: Note that

Original: LASEM shows high accuracy regardless of the base lifelong learner (e.g., HPS, TF, or DF-CNN) while introducing relatively little additional time complexity ( ∼ 30% over the base learner’s time).
Revised: LASEM shows high accuracy regardless of the base lifelong learner (e.g., HPS, TF, or DF-CNN) while introducing relatively little additional time complexity ( ∼ 30% over the base learner’s time).

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We have shown that the transfer conﬁguration can have a signiﬁcant impact on lifelong learning, and that the conﬁguration can be dynamically selected during the lifelong learning process with minimal computational cost.
Revised: We have shown that the transfer conﬁguration can have a signiﬁcant impact on lifelong learning, and that the conﬁguration can be dynamically selected during the lifelong learning process with minimal computational cost.

Original: The improvement gained by choosing the optimal transfer conﬁguration signiﬁcantly improves the performance of the
Revised: The improvement gained by choosing the optimal transfer conﬁguration signiﬁcantly improves the performance of the

Original: DF-CNN and TF over the original method and previously published results.
Revised: DF-CNN and TF over the original method and previously published results.

Original: Although we focused on layer-based transfer, LASEM could easily be extended to support partial transfer within a layer by imposing within-layer partitions and redeﬁning the transfer conﬁguration space C to support those partitions.
Revised: Although we focused on layer-based transfer, LASEM could easily be extended to support partial transfer within a layer by imposing within-layer partitions and redeﬁning the transfer conﬁguration space C to support those partitions.

Original: Discovering these partitions directly from data, or providing more ﬂexible mechanisms for partial within-layer transfer may further improve performance over the layer-based transfer we explore in this paper. 
Revised: Discovering these partitions directly from data, or providing more ﬂexible mechanisms for partial within-layer transfer may further improve performance over the layer-based transfer we explore in this paper. Acknowledgments

Revised: We would like to thank Jorge Mendez, Kyle Vedder, Meghna

Revised: Gummadi and the anonymous reviewers for their helpful feedback on this work.

Original: A. Experiment details
Revised: This research was partially supported by the Lifelong Learning Machines program from DARPA/MTO under grant #FA8750-18-2- Author Contributions SB conceived the initial idea to try varying the transfer conﬁgurations, which SL explored in the initial experiments, advised by EE and SB.

Revised: SL conceived the LASEM algorithm, which was developed in collaboration with EE.

Revised: SL and EE conceived the experiments, analyzed the results and wrote the paper, with SB providing minor edits.

Revised: A. Experiment Details

Original: This section provides detail on the experiments from the main paper.
Revised: This section provides detail on the experiments from the main paper.

Original: The experiments are based on three image recognition datasets: CIFAR-100 (Krizhevsky and Hinton, 2009), Ofﬁce-Home (Venkateswara et al., 2017) and STL-10 (Coates et al., 2011).
Revised: The experiments are based on three image recognition datasets: CIFAR-100 (Krizhevsky and Hinton, 2009), Ofﬁce-Home (Venkateswara et al., 2017) and STL- (Coates et al., 2011).

Original: CIFAR-100 consists of images of 100 classes.
Revised: CIFAR-100 consists of images of 100 classes.

Original: The lifelong learn- ing tasks are created following Lee et al.
Revised: The lifelong learning tasks are created following Lee et al.

Original: (Lee et al., 2019) by separating the dataset into ten disjoint sets of ten classes, and randomly selecting 4% of the original training data to generate training and validation sets in the ratio of 5.6:1 (170 training and 30 validation instances per task).
Revised: (Lee et al., 2019) by separating the dataset into ten disjoint sets of ten classes, and randomly selecting 4% of the original training data to generate training and validation sets in the ratio of 5.6:1 (170 training and 30 validation instances per task).

Original: The images are used without any pre-processing except re-scaling all pixel values to the range of [0 , 1] .
Revised: The images are used without any pre-processing except re-scaling all pixel values to the range of [0 , 1] .

Original: The Ofﬁce-Home dataset has images of 65 classes in four domains.
Revised: The Ofﬁce-Home dataset has images of 65 classes in four domains.

Original: Again following Lee et al.
Revised: Again following Lee et al.

Original: , lifelong learning tasks are generated by choosing ten disjoint groups of thirteen classes in two domains: Product and Real-World.
Revised: , lifelong learning tasks are generated by choosing ten disjoint groups of thirteen classes in two domains: Product and Real-World.

Original: There is no pre-deﬁned training/testing split in Ofﬁce-Home, so we randomly split the images in the ratio 6:1:3 for the training, validation, and test sets.
Revised: There is no pre-deﬁned training/testing split in Ofﬁce-Home, so we randomly split the images in the ratio 6:1:3 for the training, validation, and test sets.

Original: The image sizes are not uniform, so we resized all images to be 128-by-128 pixels and re-scaled each pixel value to the range of [0 , 1] .
Revised: The image sizes are not uniform, so we resized all images to be 128-by-128 pixels and re-scaled each pixel value to the range of [0 , 1] .

Original: We introduce a lifelong learning variant of the STL-10 dataset, which contains ten classes.
Revised: We introduce a lifelong learning variant of the STLdataset, which contains ten classes.

Original: We constructed 20 three-way classiﬁcation tasks by randomly choosing the classes, applying Gaussian noise to the images (with a mean and variance randomly sampled from {− 10% , − 5% , 0% , 5% , 10% } of the range of pixel values), and randomly swapping channels.
Revised: We constructed 20 threeway classiﬁcation tasks by randomly choosing the classes, applying Gaussian noise to the images (with a mean and variance randomly sampled from {− 10% , − 5% , 0% , 5% , 10% } of the range of pixel values), and randomly swapping channels.

Original: Note that any pair of tasks differs by at least one image class, the mean and variance of the Gaussian noise, or the order of channels for the swap.
Revised: Note that any pair of tasks differs by at least one image class, the mean and variance of the Gaussian noise, or the order of channels for the swap.

Original: We sam- pled 25% of the given training data and split it into training and validation sets with the ratio 5.7:1 (318 training and 57 validation instances per task).
Revised: We sampled 25% of the given training data and split it into training and validation sets with the ratio 5.7:1 (318 training and 57 validation instances per task).

Original: All of the original STL-10 test data are used for held-out evaluation of performance.
Revised: All of the original STL-10 test data are used for held-out evaluation of performance.

Original: The architectural details of the task models used for each data set are described in Figure 4.
Revised: The architectural details of the task models used for each data set are described in Figure 4.

Original: We used the following values for the hyper-parameters of the algorithms, following the original papers wherever possible:
Revised: We used the following values for the hyper-parameters of the algorithms, following the original papers wherever possible:

Original: • The multi-task CNN with hard parameter sharing (HPS) has no additional hyper-parameters.
Revised: • The multi-task CNN with hard parameter sharing (HPS) has no additional hyper-parameters.

Original: • Tensor factorization has a scale of the weight orthogonality constraint, whose value was chosen by grid search among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 } following the original paper (Bulat et al., 2020).
Revised: • Tensor factorization has a scale of the weight orthogonality constraint, whose value was chosen by grid search among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 } following the original paper (Bulat et al., 2020).

Original: • DF-CNN requires the size of the shared tensors and the parameters of the task-speciﬁc mappings to be speciﬁed.
Revised: • DF-CNN requires the size of the shared tensors and the parameters of the task-speciﬁc mappings to be speciﬁed.

Original: Following the original paper (Lee et al., 2019), we chose the spatial size of the shared tensors to be half the spatial size of the convolutional ﬁlters, and the spatial size of the deconvolutional ﬁlters as 3 × 3 .
Revised: Following the original paper (Lee et al., 2019), we chose the spatial size of the shared tensors to be half the spatial size of the convolutional ﬁlters, and the spatial size of the deconvolutional ﬁlters as 3 × 3 .

Original: For each convolutional layer with input channels c in and output channels c out , the number of channels in the shared tensors was one-third of c in + c out
Revised: For each convolutional layer with input channels c in and output channels c out , the number of channels in the shared tensors was one-third of c in + c out

Original: and the number of output channels of the deconvolutional ﬁlters was two-thirds of c in + c out .
Revised: and the number of output channels of the deconvolutional ﬁlters was two-thirds of c in + c out .

Original: • DEN has several regularization terms and the size of the dynamic expansion.
Revised: • DEN has several regularization terms and the size of the dynamic expansion.

Original: We used the regularization values in the authors’ published code, and set the size of the dynamic expansion to be 32 by choosing the most favorable value among { 8 , 16 , 32 , 64 } .
Revised: We used the regularization values in the authors’ published code, and set the size of the dynamic expansion to be 32 by choosing the most favorable value among { 8 , 16 , 32 , 64 } .

Original: • ProgNN requires the compression ratio of the lateral con- nections, which we set to be 2, following the original paper (Rusu et al., 2016).
Revised: • ProgNN requires the compression ratio of the lateral connections, which we set to be 2, following the original paper (Rusu et al., 2016).

Original: • For DARTS, we used the hyper-parameter settings described in the original paper (Liu et al., 2018).
Revised: • For DARTS, we used the hyper-parameter settings described in the original paper (Liu et al., 2018).

Original: A lifelong learner has access to the training data of only the current task, and it optimizes the parameters of the current task model as well as any shared knowledge, depending on the algorithm.
Revised: A lifelong learner has access to the training data of only the current task, and it optimizes the parameters of the current task model as well as any shared knowledge, depending on the algorithm.

Original: After the pre-determined number of training epochs, the task switches to a new one regardless of the convergence of the lifelong learner, which favors learners that can rapidly adapt to each task.
Revised: After the pre-determined number of training epochs, the task switches to a new one regardless of the convergence of the lifelong learner, which favors learners that can rapidly adapt to each task.

Original: When the learner encounters a new task, it initializes newly introduced parameters of the new task model, but re-uses the parameters of shared components, which initialize only once at the beginning of the ﬁrst task.
Revised: When the learner encounters a new task, it initializes newly introduced parameters of the new task model, but re-uses the parameters of shared components, which initialize only once at the beginning of the ﬁrst task.

Original: As mentioned earlier, these new task-speciﬁc parameters and shared parameters are optimized according to the training data of the new task for another batch of training epochs.
Revised: As mentioned earlier, these new task-speciﬁc parameters and shared parameters are optimized according to the training data of the new task for another batch of training epochs.

Original: We used the RMSProp optimizer with the hyper-parameter values (such as learning rate and the number of training epochs per task) described in Table 2.
Revised: We used the RMSProp optimizer with the hyper-parameter values (such as learning rate and the number of training epochs per task) described in Table 2.

Original: [Table]
Revised: We investigated several aspects of LASEM in addition to mean peak per-task accuracy. First, the catastrophic forgetting ratio is shown in Figure 5. The catastrophic forgetting ratio, proposed in (Lee et al., 2019), measures the ability of the lifelong learning algorithm to maintain its performance on previous tasks during subsequent learning.

Original: [Table]
Revised: A low ratio indicates that there is negative reverse transfer from new tasks to previously learned tasks, and so the learner experiences catastrophic forgetting. A ratio greater than 1 can be interpreted as positive backward transfer. As depicted in the ﬁgure, LASEM is able to retain the performance of previous tasks compared to transferring at all CNN layers and transferring at speciﬁc CNN-layers for all tasks (using a static transfer conﬁguration).

Original: [Table]
Revised: [Figure]

Original: [Table]
Revised: The different transfer conﬁgurations chosen by LASEM are depicted in Figure 6 for CIFAR-100 and Ofﬁce-Home. We plot the most frequent transfer conﬁgurations as well as the proportion of the time each layer was chosen to be transfer-based or task-speciﬁc. We can see that HPS tends to often prefer task-speciﬁc layers, while TF and DF-CNN are more likely to use transfer layers due to the ﬂexibility of transfer. We can also see dependence between the chosen layers, such as the DF-CNN preferring transfer among the higher layers.

Original: [Figure]
Revised: Another interesting observation is that non-tree structures, such as Alternating { 2, 4 } and sharing middle layers [0,1,1,0], are often chosen.

Original: Dataset
Revised: This contradicts the assumption of a tree structure made often by related research, and supports the consideration of more complex transfer conﬁgurations for diverse tasks. [Table] a a p h i c

Revised: F

Revised: o r g e tt i n g

Revised: R a t i o

Revised: [Table]

Revised: r o p h c F g e tt n g a

Revised: [Table]

Revised: o

Revised: [Table]

Revised: C a t a s t r o p h

Revised: i c

Revised: F

Revised: o r g e tt i n g

Revised: R a t i o

Revised: DF-CNN all DF-CNN best (top2)

Revised: LASEM DF-CNN (f) DF-CNN on Ofﬁce-Home

Revised: Dataset

Original: CIFAROfﬁce-Home
Revised: CIFAROfﬁce-Home

Original: STL
Revised: STL-

Original: [Figure]
Revised: [Table]

Original: Number

Original: [Figure] [Figure] [Table] 
Revised: [Table] [Figure]

Original: Heterogeneous Classiﬁcation

Original: Classes per

Original: [Table]
Revised: [Figure]

Original: Amount

Original: [Table]
Revised: [Figure]

Original: Data

Original: [Figure][Table]
Revised: [Figure]

Original: Ratio of Training and Validation Set [Table] [Table] [Table]
Revised: [Figure]

Original: [Table]
Revised: [Figure]

Original: Optimizer RMS Prop
Revised: L a y e r w i s e T r a n s f e r

Original: [Table] [Figure]
Revised: [Figure] [Figure]