  Neural networks notoriously suffer from the problem of catastrophic
forgetting, the phenomenon of forgetting the past knowledge when acquiring new
knowledge. Overcoming catastrophic forgetting is of significant importance to
emulate the process of "incremental learning", where the model is capable of
learning from sequential experience in an efficient and robust way.
State-of-the-art techniques for incremental learning make use of knowledge
distillation towards preventing catastrophic forgetting. Therein, one updates
the network while ensuring that the network's responses to previously seen
concepts remain stable throughout updates. This in practice is done by
minimizing the dissimilarity between current and previous responses of the
network one way or another. Our work contributes a novel method to the arsenal
of distillation techniques. In contrast to the previous state of the art, we
propose to firstly construct low-dimensional manifolds for previous and current
responses and minimize the dissimilarity between the responses along the
geodesic connecting the manifolds. This induces a more formidable knowledge
distillation with smooth properties which preserves the past knowledge more
efficiently as observed by our comprehensive empirical study.
