  The ability to learn continually without forgetting the past tasks is a
desired attribute for artificial learning systems. Existing approaches to
enable such learning in artificial neural networks usually rely on network
growth, importance based weight update or replay of old data from the memory.
In contrast, we propose a novel approach where a neural network learns new
tasks by taking gradient steps in the orthogonal direction to the gradient
subspaces deemed important for the past tasks. We find the bases of these
subspaces by analyzing network representations (activations) after learning
each task with Singular Value Decomposition (SVD) in a single shot manner and
store them in the memory as Gradient Projection Memory (GPM). With qualitative
and quantitative analyses, we show that such orthogonal gradient descent
induces minimum to no interference with the past tasks, thereby mitigates
forgetting. We evaluate our algorithm on diverse image classification datasets
with short and long sequences of tasks and report better or on-par performance
compared to the state-of-the-art approaches.
