  Several deep neural ranking models have been proposed in the recent IR
literature. While their transferability to one target domain held by a dataset
has been widely addressed using traditional domain adaptation strategies, the
question of their cross-domain transferability is still under-studied. We study
here in what extent neural ranking models catastrophically forget old knowledge
acquired from previously observed domains after acquiring new knowledge,
leading to performance decrease on those domains. Our experiments show that the
effectiveness of neuralIR ranking models is achieved at the cost of
catastrophic forgetting and that a lifelong learning strategy using a
cross-domain regularizer success-fully mitigates the problem. Using an
explanatory approach built on a regression model, we also show the effect of
domain characteristics on the rise of catastrophic forgetting. We believe that
the obtained results can be useful for both theoretical and practical future
work in neural IR.
