  Quadratic regularizers are often used for mitigating catastrophic forgetting
in deep neural networks (DNNs), but are unable to compete with recent continual
learning methods. To understand this behavior, we analyze parameter updates
under quadratic regularization and demonstrate such regularizers prevent
forgetting of past tasks by implicitly performing a weighted average between
current and previous values of model parameters. Our analysis shows the
inferior performance of quadratic regularizers arises from (a) dependence of
weighted averaging on training hyperparameters, which often results in unstable
training and (b) assignment of lower importance to deeper layers, which are
generally the cause for forgetting in DNNs. To address these limitations, we
propose Explicit Movement Regularization (EMR), a continual learning algorithm
that modifies quadratic regularization to remove the dependence of weighted
averaging on training hyperparameters and uses a relative measure for
importance to avoid problems caused by lower importance assignment to deeper
layers. Compared to quadratic regularization, EMR achieves 6.2% higher average
accuracy and 4.5% lower average forgetting.
