  Continual learning (CL) aims to enable information systems to learn from a
continuous data stream across time. However, it is difficult for existing deep
learning architectures to learn a new task without largely forgetting
previously acquired knowledge. Furthermore, CL is particularly challenging for
language learning, as natural language is ambiguous: it is discrete,
compositional, and its meaning is context-dependent. In this work, we look at
the problem of CL through the lens of various NLP tasks. Our survey discusses
major challenges in CL and current methods applied in neural network models. We
also provide a critical review of the existing CL evaluation methods and
datasets in NLP. Finally, we present our outlook on future research directions.
