  Continual learning has become increasingly important as it enables NLP models
to constantly learn and gain knowledge over time. Previous continual learning
methods are mainly designed to preserve knowledge from previous tasks, without
much emphasis on how to well generalize models to new tasks. In this work, we
propose an information disentanglement based regularization method for
continual learning on text classification. Our proposed method first
disentangles text hidden spaces into representations that are generic to all
tasks and representations specific to each individual task, and further
regularizes these representations differently to better constrain the knowledge
required to generalize. We also introduce two simple auxiliary tasks: next
sentence prediction and task-id prediction, for learning better generic and
specific representation spaces. Experiments conducted on large-scale benchmarks
demonstrate the effectiveness of our method in continual text classification
tasks with various sequences and lengths over state-of-the-art baselines. We
have publicly released our code at https://github.com/GT-SALT/IDBR.
