Original: Lifelong Learning via Online Leverage Score Sampling
Revised: Lifelong Learning via Online Leverage Score Sampling

Original: Abstract
Revised: Abstract

Original: In order to mimic the human ability of continual acquisition and transfer of knowledge across various tasks, a learning system needs the capability for life-long learning, effectively utilizing the previously acquired skills.
Revised: In order to mimic the human ability of continual acquisition and transfer of knowledge across various tasks, a learning system needs the capability for life-long learning, effectively utilizing the previously acquired skills.

Original: As such, the key challenge is to transfer and generalize the knowledge learned from one task to other tasks, avoiding interference from previous knowledge and improving the overall performance.
Revised: As such, the key challenge is to transfer and generalize the knowledge learned from one task to other tasks, avoiding interference from previous knowledge and improving the overall performance.

Original: In this paper, within the continual learning paradigm, we introduce a method that effectively forgets the less useful data samples continuously across different tasks.
Revised: In this paper, within the continual learning paradigm, we introduce a method that effectively forgets the less useful data samples continuously across different tasks.

Original: The method uses statistical leverage score information to measure the importance of the data samples in every task and adopts frequent directions approach to enable a life-long learning property.
Revised: The method uses statistical leverage score information to measure the importance of the data samples in every task and adopts frequent directions approach to enable a life-long learning property.

Original: This effectively maintains a constant training size across all tasks.
Revised: This effectively maintains a constant training size across all tasks.

Original: We ﬁrst provide some mathematical intuition for the method and then demonstrate its effectiveness with experiments on variants of
Revised: We ﬁrst provide some mathematical intuition for the method and then demonstrate its effectiveness with experiments on variants of

Original: MNIST and CIFAR100 datasets.
Revised: MNIST and CIFAR100 datasets.

Original: 1. Introduction
Revised: 1. Introduction

Original: It is a typical practice to design and optimize machine learning (ML) models to solve a single task.
Revised: It is a typical practice to design and optimize machine learning (ML) models to solve a single task.

Original: On the other hand, humans, instead of learning over isolated complex tasks, are capable of generalizing and transferring knowledge and skills learned from one task to another.
Revised: On the other hand, humans, instead of learning over isolated complex tasks, are capable of generalizing and transferring knowledge and skills learned from one task to another.

Original: This ability to remember, learn and transfer information across tasks is referred to as lifelong learning or continual learning (Thrun & Mitchell, 1995; Hassabis et al., 2017; Parisi et al., 2019).
Revised: This ability to remember, learn and transfer information across tasks is referred to as lifelong learning or continual learning (Thrun & Mitchell, 1995; Hassabis et al., 2017; Parisi et al., 2019).

Original: The major challenge for creating ML models with lifelong learning ability is that they are prone to catastrophic forgetting (McClelland et al., 1995; McCloskey & Cohen, 1989).
Revised: The major challenge for creating ML models with lifelong learning ability is that they are prone to catastrophic forgetting (McClelland et al., 1995; McCloskey & Cohen, 1989).

Original: ML models tend to forget the knowledge learned from previous tasks when re-trained on new observations corresponding to a different (but related) task.
Revised: ML models tend to forget the knowledge learned from previous tasks when re-trained on new observations corresponding to a different (but related) task.

Original: Speciﬁcally when a deep neural network (DNN) is fed with a sequence of tasks, the ability to solve the ﬁrst task will decline signiﬁcantly after training on the following tasks.
Revised: Speciﬁcally when a deep neural network (DNN) is fed with a sequence of tasks, the ability to solve the ﬁrst task will decline signiﬁcantly after training on the following tasks.

Original: The typical structure of DNNs by design does not possess the capability of preserving previously learned knowledge without interference between tasks or catastrophic forgetting.
Revised: The typical structure of DNNs by design does not possess the capability of preserving previously learned knowledge without interference between tasks or catastrophic forgetting.

Original: There have been different approaches proposed to address this issue and they can be broadly categorized in three types:
Revised: There have been different approaches proposed to address this issue and they can be broadly categorized in three types:

Original: I) Regularization : It constrains or regularizes the model parameters by adding some terms in the loss function that prevent the model from deviating signiﬁcantly from the parameters important to earlier tasks.
Revised: I) Regularization : It constrains or regularizes the model parameters by adding some terms in the loss function that prevent the model from deviating signiﬁcantly from the parameters important to earlier tasks.

Original: Typical algorithms include elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) and continual learning through synaptic intelligence (SynInt) (Zenke et al., 2017).
Revised: Typical algorithms include elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) and continual learning through synaptic intelligence (SynInt) (Zenke et al., 2017).

Original: II) Architectural modiﬁcation : It revises the model structure successively after each task in order to provide more memory and additional free parameters in the model for new task input.
Revised: II) Architectural modiﬁcation : It revises the model structure successively after each task in order to provide more memory and additional free parameters in the model for new task input.

Original: Recent examples in this direction are progressive neural networks (Rusu et al., 2016) and dynamically expanding networks (Yoon et al., 2018).
Revised: Recent examples in this direction are progressive neural networks (Rusu et al., 2016) and dynamically expanding networks (Yoon et al., 2018).

Original: III) Memory replay : It stores data samples from previous tasks in a separate memory buffer and retrains the new model based on both the new task input and the memory buffer.
Revised: III) Memory replay : It stores data samples from previous tasks in a separate memory buffer and retrains the new model based on both the new task input and the memory buffer.

Original: Popular algorithms here are gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017), incremental classiﬁer and representation learning (iCaRL) (Rebufﬁ et al., 2017).
Revised: Popular algorithms here are gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017), incremental classiﬁer and representation learning (iCaRL) (Rebufﬁ et al., 2017).

Original: Among these approaches, regularization is particularly prone to saturation of learning when the number of tasks is large.
Revised: Among these approaches, regularization is particularly prone to saturation of learning when the number of tasks is large.

Original: The additional / regularization
Revised: The additional / regularization

Original: term
Revised: term

Original: in the loss function will soon lose its competency when important parameters from different tasks are overlapped too many times.
Revised: in the loss function will soon lose its competency when important parameters from different tasks are overlapped too many times.

Original: Modiﬁcations on network architectures like progressive networks resolve the saturation issue, but do not scale as number and complexity of tasks increase.
Revised: Modiﬁcations on network architectures like progressive networks resolve the saturation issue, but do not scale as number and complexity of tasks increase.

Original: The scalability problem is also present when using memory replay and often suffer from high computational and memory costs.
Revised: The scalability problem is also present when using memory replay and often suffer from high computational and memory costs.

Original: In this paper, we propose a novel approach to lifelong learning with DNNs that addresses both the learning saturation and high computational complexity issues.
Revised: In this paper, we propose a novel approach to lifelong learning with DNNs that addresses both the learning saturation and high computational complexity issues.

Original: In this method, we progressively compresses the input information learned thus far along with the input from current task and form more efﬁciently condensed data samples.
Revised: In this method, we progressively compresses the input information learned thus far along with the input from current task and form more efﬁciently condensed data samples.

Original: The compression technique is based on the statistical leverage scores measure, and it uses frequent directions idea in order to connect the series of compression steps for a sequence of tasks.
Revised: The compression technique is based on the statistical leverage scores measure, and it uses frequent directions idea in order to connect the series of compression steps for a sequence of tasks.

Original: Our approach resembles the use of memory replay since it preserves the original input data samples from earlier tasks for further training.
Revised: Our approach resembles the use of memory replay since it preserves the original input data samples from earlier tasks for further training.

Original: However, our method does not require extra memory for training and is cost efﬁcient compared to most memory replay methods.
Revised: However, our method does not require extra memory for training and is cost efﬁcient compared to most memory replay methods.

Original: Furthermore, unlike the importance assigned to model speciﬁc parameters when using regularization methods like EWC or SynInt, we assign importance to the training data that is relevant in effectively learning new tasks, while forgetting less important information.
Revised: Furthermore, unlike the importance assigned to model speciﬁc parameters when using regularization methods like EWC or SynInt, we assign importance to the training data that is relevant in effectively learning new tasks, while forgetting less important information.

Original: 2. Online Leverage Score Sampling (OLSS)
Revised: 2. Online Leverage Score Sampling (OLSS)

Original: Before presenting the idea, let’s ﬁrst setup the problem:
Revised: Before presenting the idea, let’s ﬁrst setup the problem:

Original: Let { ( A 1 , B 1 ) , ( A 2 , B 2 ) , ..., ( A i , B i ) , ... } represent a sequence of tasks, each task consists of n i data samples and each sample has a feature dimension d and an output dimension m , i.e., input A i ∈ R n i × d and true output B i ∈ R n i × m .
Revised: Let { ( A 1 , B 1 ) , ( A 2 , B 2 ) , ..., ( A i , B i ) , ... } represent a sequence of tasks, each task consists of n i data samples and each sample has a feature dimension d and an output dimension m , i.e., input A i ∈ R n i × d and true output B i ∈ R n i × m .

Original: Here, we assume the feature and output dimensions are ﬁxed for all tasks 1 .
Revised: Here, we assume the feature and output dimensions are ﬁxed for all tasks 1 .

Original: The goal is to train a DNN over the sequence of tasks and ensure it performs well on all of them.
Revised: The goal is to train a DNN over the sequence of tasks and ensure it performs well on all of them.

Original: Here, we consider that the network’s architecture stays the same and the tasks are received in a sequential manner.
Revised: Here, we consider that the network’s architecture stays the same and the tasks are received in a sequential manner.

Original: Formally, with f representing a DNN, our objective is to minimize the loss 2 : min f (cid:13)(cid:13)
Revised: Formally, with f representing a DNN, our objective is to minimize the loss 2 : min f (cid:13)(cid:13)

Original: f ( A ) − B (cid:13)(cid:13) 2 2 where
Revised: f ( A ) − B (cid:13)(cid:13) 2 2 where

Original: A =  
Revised: A =  

Original: A
Revised: A

Original: A...
Revised: A...

Original: A i ...
Revised: A i ...

Original:   and B =  
Revised:   and B =  

Original: B
Revised: B

Original: B... B i ...
Revised: B... B i ...

Original:   .
Revised:   .

Original: Under this setup, let’s look at some existing models:
Revised: Under this setup, let’s look at some existing models:

Original: Online EWC trains f on task ( A i , B i ) with a loss function containing additional penalty terms min f (cid:13) (cid:13) f
Revised: Online EWC trains f on task ( A i , B i ) with a loss function containing additional penalty terms min f (cid:13) (cid:13) f

Original: [Equation]
Revised: [Equation]

Original: A i )
Revised: A i )

Original: [Equation]
Revised: [Equation]

Original: (cid:80) i − 1 j =1 Λ j and each Λ j is deﬁned as the change of important parameters (using Fisher information matrix) in f with respect to the j th task.
Revised: (cid:80) i − 1 j =1 Λ j and each Λ j is deﬁned as the change of important parameters (using Fisher information matrix) in f with respect to the j th task.

Original: GEM keeps an extra memory buffer containing data samples from each of the previous tasks M k with k <
Revised: GEM keeps an extra memory buffer containing data samples from each of the previous tasks M k with k <

Original: i , it trains on the current task ( A i , B i ) with a regular loss function min (cid:13)(cid:13)
Revised: i , it trains on the current task ( A i , B i ) with a regular loss function min (cid:13)(cid:13)

Original: f
Revised: f

Original: ( A i ) − B i (cid:13)(cid:13) 2 2 , but subject to inequalities on each update of f , (cid:104) ∂ (cid:107)
Revised: ( A i ) − B i (cid:13)(cid:13) 2 2 , but subject to inequalities on each update of f , (cid:104) ∂ (cid:107)

Original: f θ ( A i ) − B i (cid:107) 2 2 ∂θ ,
Revised: f θ ( A i ) − B i (cid:107) 2 2 ∂θ ,

Original: [Table]
Revised: [Table]

Original: (cid:107)
Revised: (cid:107)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:105) ≥ 

Revised: (cid:105) ≥

Original: for all k < i .
Revised: for all k < i .

Original: The new approach OLSS is to ﬁnd an approximation of A in a streaming manner, i.e., to form an ˆ A i to approximate
Revised: The new approach OLSS is to ﬁnd an approximation of A in a streaming manner, i.e., to form an ˆ A i to approximate

Original: [ A 1 A 2 · · · A i ] T such that the resulting
Revised: [ A 1 A 2 · · · A i ] T such that the resulting

Original: [Equation]
Revised: [Equation]

Original: is likely to perform on all tasks as good as
Revised: is likely to perform on all tasks as good as

Original: [Equation]
Revised: [Equation]

Original: To avoid extra memory and computation cost during the training process, we restrict the approximate ˆ A i to have the same number of rows as the current task A i .
Revised: To avoid extra memory and computation cost during the training process, we restrict the approximate ˆ A i to have the same number of rows as the current task A i .

Original: Equation (1) and (2) represent nonlinear least squares problems.
Revised: Equation (1) and (2) represent nonlinear least squares problems.

Original: It is to be noted that a nonlinear least squares problem can be solved with an approximation deduced from an iteration of linear least squares problems with
Revised: It is to be noted that a nonlinear least squares problem can be solved with an approximation deduced from an iteration of linear least squares problems with

Original: J T J ∆ θ = J T ∆ B where J is the Jacobian of f at each update (Gauss-Newton Method).
Revised: J T J ∆ θ = J T ∆ B where J is the Jacobian of f at each update (Gauss-Newton Method).

Original: Besides this technique, there are various approaches in addressing this problem.
Revised: Besides this technique, there are various approaches in addressing this problem.

Original: Here we adopt a cost effective simple randomization technique - leverage score sampling, which has been used extensively in solving large scale linear least squares and low rank approximation problems (Woodruff, 2014; Cohen et al., 2017).
Revised: Here we adopt a cost effective simple randomization technique - leverage score sampling, which has been used extensively in solving large scale linear least squares and low rank approximation problems (Woodruff, 2014; Cohen et al., 2017).

Original: Statistical Leverage Score
Revised: Statistical Leverage Score

Original: Deﬁnition 1 (Drineas et al., 2012)
Revised: Deﬁnition 1 (Drineas et al., 2012)

Original: Given a matrix A ∈
Revised: Given a matrix A ∈

Original: R n × d with n > d , let U denote the n × d matrix consisting of the d left singular vectors of A , and let U ( i, :) denote the i -th row of U , then the statistical leverage score of the i -th row of A is deﬁned as (cid:13)(cid:13)(cid:13)
Revised: R n × d with n > d , let U denote the n × d matrix consisting of the d left singular vectors of A , and let U ( i, :) denote the i -th row of U , then the statistical leverage score of the i -th row of A is deﬁned as (cid:13)(cid:13)(cid:13)

Original: U
Revised: U

Original: ( i, :) (cid:13) (cid:13)(cid:13) 2 2 for i ∈ { 1 , ..., n } .
Revised: ( i, :) (cid:13) (cid:13)(cid:13) 2 2 for i ∈ { 1 , ..., n } .

Original: Statistical leverage scores deﬁne the relevant non-uniformity structure of a matrix and a higher score indicates a heavier weight of the row contributing to the non-uniformity of the matrix; it has been widely used for constructing a randomized sketch of a matrix (Drineas et al., 2012; Woodruff, 2014).
Revised: Statistical leverage scores deﬁne the relevant non-uniformity structure of a matrix and a higher score indicates a heavier weight of the row contributing to the non-uniformity of the matrix; it has been widely used for constructing a randomized sketch of a matrix (Drineas et al., 2012; Woodruff, 2014).

Original: In our case, given an input matrix A , we will compute the leverage score of each row, then sample the rows with probability proportional to the scores.
Revised: In our case, given an input matrix A , we will compute the leverage score of each row, then sample the rows with probability proportional to the scores.

Original: Using leverage score sampling, we are able to select the important samples given a dataset.
Revised: Using leverage score sampling, we are able to select the important samples given a dataset.

Original: The remaining problem is to embed it in a sequence of tasks.
Revised: The remaining problem is to embed it in a sequence of tasks.

Original: In order to achieve this, we make use of the concept of frequent directions.
Revised: In order to achieve this, we make use of the concept of frequent directions.

Original: Frequent Directions
Revised: Frequent Directions

Original: Frequent directions extends the idea of frequent items in item frequency approximation problem to a matrix (Liberty, 2013; Ghashami et al., 2016; Teng & Chu, 2018).
Revised: Frequent directions extends the idea of frequent items in item frequency approximation problem to a matrix (Liberty, 2013; Ghashami et al., 2016; Teng & Chu, 2018).

Original: Given a matrix A ∈ R n × d whose rows are received one by one and a space parameter (cid:96) , the algorithm considers the ﬁrst 2 (cid:96) rows in A and shrinks its top (cid:96) orthogonal vectors by the same amount to obtain an (cid:96) × d matrix; then combines them with the next (cid:96) rows in A for the next iteration, repeat the procedure until reaching the ﬁnal sketch of dimension (cid:96) × d .
Revised: Given a matrix A ∈ R n × d whose rows are received one by one and a space parameter (cid:96) , the algorithm considers the ﬁrst 2 (cid:96) rows in A and shrinks its top (cid:96) orthogonal vectors by the same amount to obtain an (cid:96) × d matrix; then combines them with the next (cid:96) rows in A for the next iteration, repeat the procedure until reaching the ﬁnal sketch of dimension (cid:96) × d .

Original: Frequent directions algorithm is targeted at ﬁnding a low rank approximation on a continuously expanding matrix.
Revised: Frequent directions algorithm is targeted at ﬁnding a low rank approximation on a continuously expanding matrix.

Original: This is well suited for a continuous stream of data (tasks) within the lifelong learning setting.
Revised: This is well suited for a continuous stream of data (tasks) within the lifelong learning setting.

Original: We present the step by step procedure of performing leverage score sampling together with compression using frequent directions idea in
Revised: We present the step by step procedure of performing leverage score sampling together with compression using frequent directions idea in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: In our setting, we append the new task data samples to the existing buffer set and perform leverage score sampling to form a new buffer set and then train on it, this process is repeated for the entire sequence of tasks.
Revised: In our setting, we append the new task data samples to the existing buffer set and perform leverage score sampling to form a new buffer set and then train on it, this process is repeated for the entire sequence of tasks.

Original: Main Algorithm
Revised: Main Algorithm

Original: Algorithm
Revised: Algorithm

Original: OLSS
Revised: OLSS

Original: Input : A sequence of tasks { ( A 1 , B 1 ) , ..., ( A i , B i ) , ... } with A i ∈ R n i × d and B i ∈ R n i × m ; initialization of the model parameters; a space parameter (cid:96) i.e., number of samples to pass in the model for training.
Revised: Input : A sequence of tasks { ( A 1 , B 1 ) , ..., ( A i , B i ) , ... } with A i ∈ R n i × d and B i ∈ R n i × m ; initialization of the model parameters; a space parameter (cid:96) i.e., number of samples to pass in the model for training.

Original: It can be set as n i or even smaller after receiving the i -th task, which avoids extra memory and computations during training.
Revised: It can be set as n i or even smaller after receiving the i -th task, which avoids extra memory and computations during training.

Original: Output : A trained neural network on a sequence of tasks.
Revised: Output : A trained neural network on a sequence of tasks.

Original: Step 1 Initialize a buffer set S
Revised: Step 1 Initialize a buffer set S

Original: = { ˆ A, ˆ B } where both ˆ A and ˆ B are empty.
Revised: = { ˆ A, ˆ B } where both ˆ A and ˆ B are empty.

Original: Step 2 While the i th task is presented:
Revised: Step 2 While the i th task is presented:

Original: Step
Revised: Step

Original: If ˆ A and ˆ B are empty:
Revised: If ˆ A and ˆ B are empty:

Original: Step set ˆ A
Revised: Step set ˆ A

Original: =
Revised: =

Original: A i and ˆ B = B i ,
Revised: A i and ˆ B = B i ,

Original: Step
Revised: Step

Original: else:
Revised: else:

Original: Stepset ˆ A = (cid:34) ˆ A
Revised: Stepset ˆ A = (cid:34) ˆ A

Original: A i (cid:35)
Revised: A i (cid:35)

Original: and ˆ B = (cid:34) ˆ B B
Revised: and ˆ B = (cid:34) ˆ B B

Original: i
Revised: i

Original: Randomly select (cid:96) rows of ˆ A and ˆ B without replacement based on probability (cid:13)(cid:13)
Revised: Randomly select (cid:96) rows of ˆ A and ˆ B without replacement based on probability (cid:13)(cid:13)

Original: U j, : (cid:13)(cid:13) 2 2 / (cid:107) U (cid:107) 2 F for j ∈ { 1 , ..., n i + (cid:96) } (or j ∈ { 1 , ..., n i } when i = 1 ) and set them as ˆ A and ˆ B respectively.
Revised: U j, : (cid:13)(cid:13) 2 2 / (cid:107) U (cid:107) 2 F for j ∈ { 1 , ..., n i + (cid:96) } (or j ∈ { 1 , ..., n i } when i = 1 ) and set them as ˆ A and ˆ B respectively.

Original: When n i is large, the SVD (singular value decomposition) of matrix ˆ A ∈ R ( n i + (cid:96) )
Revised: When n i is large, the SVD (singular value decomposition) of matrix ˆ A ∈ R ( n i + (cid:96) )

Original: ×
Revised: ×

Original: d in Step 6 is computationally expensive, we could use a streaming SVD method to speed up the process if (cid:96) is chosen much smaller than n i .
Revised: d in Step 6 is computationally expensive, we could use a streaming SVD method to speed up the process if (cid:96) is chosen much smaller than n i .

Original: In that case the computational cost for SVD could be reduced from O (( n i + (cid:96) )
Revised: In that case the computational cost for SVD could be reduced from O (( n i + (cid:96) )

Original: d 2 ) to O (log 2 ( n i + (cid:96) ) (cid:96)d 2 ) (assuming d < (cid:96) < n i ).
Revised: d 2 ) to O (log 2 ( n i + (cid:96) ) (cid:96)d 2 ) (assuming d < (cid:96) < n i ).

Original: In addition, there exists various efﬁcient ways to approximate the leverage scores (Drineas et al., 2012; Rudi et al., 2018) which would further reduce the computational cost.
Revised: In addition, there exists various efﬁcient ways to approximate the leverage scores (Drineas et al., 2012; Rudi et al., 2018) which would further reduce the computational cost.

Original: Remark: A major concern with this algorithm is that leverage scores is a linear measure, i.e., the selected samples capture the important information embedded linearly in the data matrix which may not fully represent the importance of the data samples.
Revised: Remark: A major concern with this algorithm is that leverage scores is a linear measure, i.e., the selected samples capture the important information embedded linearly in the data matrix which may not fully represent the importance of the data samples.

Original: Another related issue is that the nonlinear information probably depend on the structure of f , the DNN.
Revised: Another related issue is that the nonlinear information probably depend on the structure of f , the DNN.

Original: As such, there may be some underlying dependency of a data sample’s importance on the DNN architecture.
Revised: As such, there may be some underlying dependency of a data sample’s importance on the DNN architecture.

Original: We leave this open as a future research direction.
Revised: We leave this open as a future research direction.

Original: 3. Experiments
Revised: 3. Experiments

Original: We evaluate the performance of
Revised: We evaluate the performance of

Original: the proposed algorithm OLSS on three classiﬁcation tasks used as benchmarks in related prior work.
Revised: the proposed algorithm OLSS on three classiﬁcation tasks used as benchmarks in related prior work.

Original: • Rotated MNIST (Lopez-Paz & Ranzato, 2017): a variant of the MNIST dataset of handwriten digits (LeCun et al., 1998), the digits in each task are rotated by a ﬁxed angle between 0 ◦ to 180 ◦ .
Revised: • Rotated MNIST (Lopez-Paz & Ranzato, 2017): a variant of the MNIST dataset of handwriten digits (LeCun et al., 1998), the digits in each task are rotated by a ﬁxed angle between 0 ◦ to 180 ◦ .

Original: The experiment is on 20 tasks and each task consists of 60 , 000 training and 10 , 000 testing samples.
Revised: The experiment is on 20 tasks and each task consists of 60 , 000 training and 10 , 000 testing samples.

Original: • Permutated MNIST (Kirkpatrick et al., 2017): a variant of the MNIST dataset (LeCun et al., 1998), the digits in each task are transformed by a ﬁxed permutation of pixels.
Revised: • Permutated MNIST (Kirkpatrick et al., 2017): a variant of the MNIST dataset (LeCun et al., 1998), the digits in each task are transformed by a ﬁxed permutation of pixels.

Original: The experiment is on 20 tasks and each task consists of 60 , 000 training and 10 , 000 testing samples.
Revised: The experiment is on 20 tasks and each task consists of 60 , 000 training and 10 , 000 testing samples.

Original: • Incremental CIFAR

Revised: • Incremental CIFAR

Original: (Rebufﬁ et al., 2017; Zenke et al., 2017): a variant of the CIFAR object recognition dataset with 100 classes (Krizhevsky, 2009).
Revised: (Rebufﬁ et al., 2017; Zenke et al., 2017): a variant of the CIFAR object recognition dataset with 100 classes (Krizhevsky, 2009).

Original: The experiment is on 20 tasks and each task consists ofclasses; each task consists of 2 , 500 training andtesting samples.
Revised: The experiment is on 20 tasks and each task consists ofclasses; each task consists of 2 , 500 training andtesting samples.

Original: Where, each task introduces a new set of classes; for a total number of 20 tasks, each new task concerns examples from a disjoint subset of 5 classes.
Revised: Where, each task introduces a new set of classes; for a total number of 20 tasks, each new task concerns examples from a disjoint subset of 5 classes.

Original: In the original setting of (Lopez-Paz & Ranzato, 2017) a softmax layer is added to the output vector which only allows entries representing the 5 classes in the current task to output values larger than 0 .
Revised: In the original setting of (Lopez-Paz & Ranzato, 2017) a softmax layer is added to the output vector which only allows entries representing the 5 classes in the current task to output values larger than 0 .

Original: In our setting, we allow the entries representing all the past occurring classes to output values larger than 0 .
Revised: In our setting, we allow the entries representing all the past occurring classes to output values larger than 0 .

Original: We believe this is a more natural setup for lifelong learning
Revised: We believe this is a more natural setup for lifelong learning

Original: The DNN used for rotated and permuted MNIST is an MLP with 2 hidden layers and each with 400 units; whereas a
Revised: The DNN used for rotated and permuted MNIST is an MLP with 2 hidden layers and each with 400 units; whereas a

Original: ResNet18 is used for the incremental CIFAR100 experiment.
Revised: ResNet18 is used for the incremental CIFAR100 experiment.

Original: We train 5 epochs with batch size 200 on rotated and permuted MNIST datasets and 10 epochs with batch sizeon incremental CIFAR100.
Revised: We train 5 epochs with batch size 200 on rotated and permuted MNIST datasets and 10 epochs with batch sizeon incremental CIFAR100.

Original: In all experiments we compare the following algorithms: I)
Revised: In all experiments we compare the following algorithms: I)

Original: A simple SGD predictor, II)
Revised: A simple SGD predictor, II)

Original: EWC (Kirkpatrick et al., 2017), III) GEM (Lopez-Paz &
Revised: EWC (Kirkpatrick et al., 2017), III) GEM (Lopez-Paz &

Original: Ranzato, 2017) and IV) OLSS (ours).
Revised: Ranzato, 2017) and IV) OLSS (ours).

Original: In all the algorithms, we use a plain SGD optimizer.
Revised: In all the algorithms, we use a plain SGD optimizer.

Original: All algorithms were implemented based on the publicly available code from the original authors of the GEM paper (Lopez-Paz & Ranzato, 2017).
Revised: All algorithms were implemented based on the publicly available code from the original authors of the GEM paper (Lopez-Paz & Ranzato, 2017).

Original: The regularization and memory hyper-parameters in EWC and GEM were set as described in (Lopez-Paz & Ranzato, 2017).
Revised: The regularization and memory hyper-parameters in EWC and GEM were set as described in (Lopez-Paz & Ranzato, 2017).

Original: The space parameter for our OLSS algorithm was set to be equal to the number of samples in each task; the learning rate for each algorithm was determined through a grid search on { 0 . 001 , 0 . 003 , 0 . 01 , 0 .
Revised: The space parameter for our OLSS algorithm was set to be equal to the number of samples in each task; the learning rate for each algorithm was determined through a grid search on { 0 . 001 , 0 . 003 , 0 . 01 , 0 .

Original: Results
Revised: Results

Original: Comparing across all the algorithms, we summarize the average test accuracy on the learned tasks in Figure 1 (see
Revised: Comparing across all the algorithms, we summarize the average test accuracy on the learned tasks in Figure 1 (see

Original:  Figure S1 in the supplemental material for the change in the test accuracy at the ﬁrst task, as more tasks are learned.) and the computational costs for each algorithm in Table 1.
Revised: Appendix Figure 2 for the change in the test accuracy at the ﬁrst task, as more tasks are learned.) and the computational costs for each algorithm in Table 1.

Original: As observed from the ﬁgures, across the three benchmarks, 
Revised: As observed from the ﬁgures, across the three benchmarks, OLSS and

Original: OLSS and GEM achieve similar accuracy and signiﬁcantly outperform both EWC and simple SGD training.
Revised: GEM achieve similar accuracy and signiﬁcantly outperform both EWC and simple SGD training.

Original: Nevertheless, GEM demands much higher computational resources (see Table 1) as the algorithm requires a constraint validation step and a potential gradient projection step to correct for constraint violations across all previously learned tasks during training (see Section 3 in (Lopez-Paz & Ranzato, 2017)).
Revised: Nevertheless, GEM demands much higher computational resources (see Table 1) as the algorithm requires a constraint validation step and a potential gradient projection step to correct for constraint violations across all previously learned tasks during training (see Section 3 in (Lopez-Paz & Ranzato, 2017)).

Original: In detail, for GEM, the time complexity is proportional to the product of the number of samples kept in the memory buffer, the number of parameters in the model and the number of iterations required to converge.
Revised: In detail, for GEM, the time complexity is proportional to the product of the number of samples kept in the memory buffer, the number of parameters in the model and the number of iterations required to converge.

Original: In contrast, 
Revised: In contrast, OLSS requires a

Original: OLSS requires a SVD (or QR factorization) to compute the leverage scores for each task which can be achieved in a time complexity proportional to the product of the square of the number of features and the number of data samples, and is much less compared to GEM.
Revised: SVD (or QR factorization) to compute the leverage scores for each task which can be achieved in a time complexity proportional to the product of the square of the number of features and the number of data samples, and is much less compared to GEM.

Original: As observed in  Figure S1 of the supplemental material, OLSS shows robustness to catastrophic forgetting of the ﬁrst task with positive backward transfer across all three datasets while learning the remaining sequence of tasks.
Revised: As observed in Appendix Figure 2, OLSS shows robustness to catastrophic forgetting of the ﬁrst task with positive backward transfer across all three datasets while learning the remaining sequence of tasks.

Original: In the case of rotated and permuted MNIST, OLSS is the most robust method.
Revised: In the case of rotated and permuted MNIST, OLSS is the most robust method.

Original: As presented in  Figure S2 of the supplemental material, after training on the whole sequence of tasks, both GEM and OLSS are able to preserve the accuracy for most tasks on rotated and permuted MNIST.
Revised: As presented in Appendix Figure 3, after training on the whole sequence of tasks, both GEM and OLSS are able to preserve the accuracy for most tasks on rotated and permuted MNIST.

Original: In contrast, it is hard to preserve the accuracy of the previously trained tasks on CIFAR100 for all algorithms.
Revised: In contrast, it is hard to preserve the accuracy of the previously trained tasks on CIFAR100 for all algorithms.

Original: As we noted earlier, EWC ex-
Revised: As we noted earlier, EWC exhibits a saturation issue

Original: [Figure]
Revised: [Figure]

Original: hibits a saturation issue when the number of tasks increases.
Revised: when the number of tasks increases.

Original: This may hold for most regularization methods in order to achieve continual learning, as they target constraining the model parameters successively, thereby limiting the model capacity.
Revised: This may hold for most regularization methods in order to achieve continual learning, as they target constraining the model parameters successively, thereby limiting the model capacity.

Original: [Table]
Revised: [Table]

Original: 4. Conclusions
Revised: 4. Conclusions

Original: We presented a new approach in addressing the lifelong learning problem with deep neural networks.
Revised: We presented a new approach in addressing the lifelong learning problem with deep neural networks.

Original: It is inspired by the randomization and compression techniques typically used in statistical analysis.
Revised: It is inspired by the randomization and compression techniques typically used in statistical analysis.

Original: We combined a simple importance sampling technique - leverage score sampling with the frequent directions concept and developed an online effective forgetting or compression mechanism that enables lifelong learning across a sequence of tasks.
Revised: We combined a simple importance sampling technique - leverage score sampling with the frequent directions concept and developed an online effective forgetting or compression mechanism that enables lifelong learning across a sequence of tasks.

Original: Despite its simple structure, the results on MNIST and CIFAR100 experiments show its effectiveness as compared to recent state of the art. 
Revised: Despite its simple structure, the results on MNIST and CIFAR100 experiments show its effectiveness as compared to recent state of the art. A. Further experimental results [Figure] [Figure]

Revised: [Table]