  Continual learning of new knowledge over time is one desirable capability for
intelligent systems to recognize more and more classes of objects. Without or
with very limited amount of old data stored, an intelligent system often
catastrophically forgets previously learned old knowledge when learning new
knowledge. Recently, various approaches have been proposed to alleviate the
catastrophic forgetting issue. However, old knowledge learned earlier is
commonly less preserved than that learned more recently. In order to reduce the
forgetting of particularly earlier learned old knowledge and improve the
overall continual learning performance, we propose a simple yet effective
fusion mechanism by including all the previously learned feature extractors
into the intelligent model. In addition, a new feature extractor is included to
the model when learning a new set of classes each time, and a feature extractor
pruning is also applied to prevent the whole model size from growing rapidly.
Experiments on multiple classification tasks show that the proposed approach
can effectively reduce the forgetting of old knowledge, achieving
state-of-the-art continual learning performance.
