Paper:
Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution -that of using experience replay buffers for all past events -with a mixture of on-and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

Review:
This paper proposes a particular variant of experience replay with behavior cloning as a method for continual learning. The approach achieves good performance while not requiring a task label. This paper makes the point that I definitely agree with that all of the approaches being considered should compare to experience replay and that in reality many of them rarely do better. However, I am not totally convinced when it comes to the value of the actual novel aspects of this paper. 
Much of the empirical analysis of experience replay (i.e. the buffer size, the ratio of past and novel experiences, etc…) was not surprising or particular novel in my eyes. The idea of using behavior cloning is motivated fully through the lens of catastrophic forgetting and promoting stability and does not at all address achieving plasticity. This was interesting to me as the authors do mention the stability-plasticity dilemma, but a more theoretical analysis of why behavior cloning is somehow the right method among various choices to promote stability while not sacrificing or improving plasticity was definitely missing for me. Other options can certainly be considered as well if your aim is just to add stability to experience replay such a notion of weight importance for the past like in EwC (Kirkpatric et al., 2017) and many other papers or using knowledge distillation like LwF (Li and Hoeim, 2016). LwF in particular seems quite related. I wonder how LwF + experience replay compares to the approach proposed here. In general the discourse could become a lot strong in my eyes if it really considered various alternatives and explained why behavior cloning provides theoretical value. 
Overall, behavior cloning seems to help a little bit based on the experiments provided, but this finding is very likely indicative of the particular problem setting and seemingly not really a game changer. In the paper, they explore settings with fairly prolonged periods of training in each RL domain one at a time. If the problem was to become more non-stationary with more frequent switching (i.e. more in line with the motivation of lifelong learning), I would imagine that increasing stability is not necessarily a good thing and may slow down future learning.