Regularization Shortcomings for Continual Learning
Authors: Timothee LESORT, Andrei Stoian
Keywords: Continual Learning, Regularization
Abstract: In most machine learning algorithms, training data is assumed to be independent and identically distributed (iid).  
When it is not the case, the performances of the algorithms are challenged, leading to the famous phenomenon of \textit{catastrophic forgetting}. Algorithms dealing with it are gathered in the \textit{Continual Learning} research field. In this paper, we study the \textit{regularization} based approaches to continual learning and show that those approaches can not learn to discriminate classes from different tasks in an elemental continual benchmark, the class-incremental setting.
We make theoretical reasoning to prove this shortcoming and illustrate it with experiments.
Moreover, we show that it can have some important consequences on multi-tasks reinforcement learning or in pre-trained models used for continual learning.
We believe this paper to be the first to propose a theoretical description of regularization shortcomings for continual learning.