One major obstacle towards artificial intelligence is the poor ability of models to quickly solve new problems, without forgetting previously acquired knowledge. To better understand this issue, we study the problem of continual learning, where the model observes, once and one by one, examples concerning a sequence of tasks. First, we propose a set of metrics to evaluate models learning over a continuum of data. These metrics characterize models not only by their test accuracy, but also in terms of their ability to transfer knowledge across tasks. Second, we propose a model for continual learning, called Gradient of Episodic Memory (GEM), which alleviates forgetting while allowing beneficial transfer of knowledge to previous tasks. Our experiments on variants of MNIST and CIFAR-100 demonstrate the strong performance of GEM when compared to the state-of-the-art. 1. Non-iid input data: the continuum of data is not iid with respect to any fixed probability distribution P (X, T, Y ) since, once tasks switch, a whole sequence of examples from the new task may be observed. 2. Catastrophic forgetting: learning new tasks may hurt the performance of the learner at previously solved tasks. 3. Transfer learning: when the tasks in the continuum are related, there exists an opportunity for transfer learning. This would translate into a faster learning of new tasks, and performance improvements in old tasks. The rest of this paper is organized as follows. In Section 2, we formalize the problem of continual learning, and introduce the metrics necessary to evaluate learners in this scenario. In Section 3, we propose GEM, a model to learn over continuums of data that alleviates forgetting, while transferring beneficial knowledge to past tasks. In Section 4, we compare the performance of GEM to the stateof-the-art. Finally, we conclude by reviewing the related literature in Section 5, and offer some directions for future research in Section 6. The central object of study in this paper is the continuum of data of eq. (1), where each triplet (x i , t i , y i ) is formed by a feature vector x i ∈ X ti , a task descriptor t i ∈ T , and a target vector y i ∈ Y ti . For simplicity, we assume that the continuum is locally iid, that is, every triplet ( While observing data as in eq. (1) example by example, our goal is to learn a predictor f : X × T → Y, which can be queried at any time to predict the target vector associated to a test pair (x, t). Such test pair can belong to a task that we have observed in the past, the current task, or a task that we will experience (or not) in the future. An important component in our framework is the collection of task descriptors t 1 , . . . , t n ∈ T . In the simplest case, the task descriptors are integers, t i = i ∈ Z, enumerating the different tasks appearing in the continuum of data. More generally, task descriptors t i could be structured objects, such as a paragraph of natural language explaining how to solve the i-th task. Rich task descriptors offer an opportunity for zero-shot learning, since the relation between tasks could be inferred using task descriptors alone. Furthermore, task descriptors disambiguate similar learning tasks. In particular, the same input x i could appear in two different tasks, one requiring classification, and the other requiring regression. Task descriptors can reference the existence of multiple learning environments, or provide additional (possibly hierarchical) contextual information about each of the examples. Next, we discuss the training protocol and evaluation metrics for continual learning. Most of the literature about learning over a sequence of tasks [Rusu et al., 2016 , Fernando et al., 2017 , Kirkpatrick et al., 2017 , Rebuffi et al., 2017 describes a setting where i) the number of tasks is small, ii) the number of examples per task is large, iii) the learner performs several epochs over the examples concerning each task, and iv) the only metric reported is the average performance across all tasks. In contrast, we are interested in the "more human-like" setting where i) the number of tasks is large, ii) the number of training examples per task is small, iii) the learner observes the examples concerning each task only once, and iv) we report metrics that measure both transfer and forgetting. Therefore, at training time we provide the learner with only one example at the time (or a small mini-batch), in the form of a triplet (x i , t i , y i ). The learner never experiences the same example twice, and tasks are streamed in sequence. 1 Besides monitoring its performance across tasks, it is also important to assess the ability of the learner to transfer knowledge. More specifically, we would like to measure: 1. Backward transfer (BWT), which is the influence that learning a task t has on the performance on a previous task t ′ ≺ t. On the one hand, there exists positive backward transfer
A special mention here goes to Gradient Episodic Memory for Continual Learning REF , as it moves a step forward towards the online setting: it assumes that the learner receives examples one by one but simplifies the scenario to locally i.i.d. drawn samples from a task distribution.
Gradient Episodic Memory for Continual Learning
NIPS
Computer Science Mathematics