Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives (2022)

Abstract: In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind the training instability, where models learn and forget different negative groups during training iterations. We then propose ANCE-Tele, which accumulates momentum negatives from past iterations and approximates future iterations using lookahead negatives, as “teleportations” along the time axis to smooth the learning process. On web search and OpenQA, ANCE-Tele outperforms previous state-of-the-art systems of similar size, eliminates the dependency on sparse retrieval negatives, and is competitive among systems using significantly more (50x) parameters. Our analysis demonstrates that teleportation negatives reduce catastrophic forgetting and improve convergence speed for dense retrieval training. The source code of this paper is available at https://github.com/OpenMatch/ANCE-Tele.

## introduction dense retrieval (dr) learns to represent data into a continuous representation space and matches user information needs ("query") with target information ("document") via efficient nearest neighbor search @xcite . recent research shows strong empirical advantages of dense retrieval in various information access scenarios, such as openqa @xcite , web search @xcite , and conversational search @xcite . a unique challenge of dense retrieval is in the selection of training negatives @xcite . for each query, dense retrieval models need to distinguish a few relevant documents from all other negative documents in the entire corpus, the latter often beyond the scale of millions, making negative sampling a necessity. at the same time, the nature of retrieval makes random negatives trivial and uninformative @xcite , making effective negative sampling difficult. recent research addressed this challenge using an iterative training approach: first training using negatives generated by sparse retrieval for a while, then refreshing the training negatives using the beingtrained dr models themselves, and alternating the two phases till convergence @xcite . the iterative training-and-negative-refreshing approach yields strong results and became a standard in many dr systems @xcite @xcite . however, the refresh of hard negatives may change the learning landscape too dramatically and cause optimization issues. many found the little benefit of training more than one refresh @xcite and the significant fluctuations in the model accuracy with more iterations due to training instability @xcite . in this paper, we aim to address the instability issue in dense retrieval training. we first conduct a deep investigation on the model behaviors during training and reveal a phenomenon behind the instability: catastrophic forgetting @xcite . after training with refreshed negatives in one episode, the dense retriever's accuracy drops on a large fraction of training queries (20-30%) that have been learned earlier. our analysis shows that negatives refreshed in different iterations may differ drastically, e.g., covering distinct distractions, and model training "swings" between these distractor groups, rather than capturing them all together. with these observations, we develop ance-tele, which upgrades ance @xcite , a standard dense retrieval training strategy, using teleportation negatives with momentum and lookahead mechanisms. specifically, momentum negatives record the negatives encountered in past iterations, while lookahead negatives use the neighbors of positive documents as surrogates to approximate the negatives appearing in the future. these teleportation negatives smooth out the training process of dense retrieval. figure 1 illustrates the training process of ance-tele. in our experiments on web search and openqa, ance-tele outperforms previous state-of-the-art systems of similar parameter sizes, without combining negatives from sparse retrieval or distillation from cross-encoder teachers. simply including the teleportation negatives in training enables a bertbase sized dense retriever (110 million parameters) to outperform the recent state-of-the-art model with five billion parameters. our analysis on the learning dynamics of ance-tele confirmed the benefits of teleportation negatives. the momentum negatives from previous iterations reduce the catastrophic forgetting issue and improve learning stability, as they note the model to "remember" previously learned signals. the lookahead negatives sampled from the neighborhood of the positive documents provide an efficient forecast of future hard negatives and also improve the convergence speed. the two together improve the stability, efficiency, and effectiveness of dense retrieval training, while imposing zero gpu computation overhead. after recapping related work, we investigate the instability issue of the iterative training process in section 3 and present ance-tele in section 4. experimental settings and evaluations are discussed in section 5 and section 6. we conclude in section 7. ## related work the classic idea of learning representations for retrieval @xcite is recently revived with pretrained language models @xcite . @xcite first presented the bert-based dual-encoder dense retrieval formulation. @xcite introduced bm25 negatives in dr training. @xcite showed the necessity of hard negatives and introduced the iterative training-and-negative-refreshing process. these techniques formed the basic dense retrieval setup that achieves strong performances on a wide range of scenarios. to improve dense retrieval training strategy is an active research front. @xcite mitigated the risk of the delayed negative refresh by mining "real-time" hard negatives with a step-wise updating query encoder and a fixed document index. @xcite observed that the iterations may lead to a fragile local optimal and balanced the negatives among query clusters. @xcite filtered the hard negatives using a stronger cross-encoder ranking model. with access to querydocument term interactions, the ranking models are more powerful and can enhance dr training via knowledge distillation as well @xcite @xcite . @xcite trained a series of dense retrievers through boosting and improved retrieval accuracy under approximate nearest neighbor search (ann). recent research also identified several mismatches between pretrained models and dr. one mismatch is the locality of token level pretraining versus the needs of full sequence embeddings in dr @xcite , which can be reduced by enforcing an information bottleneck on the sequence embedding in pretraining @xcite . another is the lack of alignment in the pretrained sequence representations, which can be improved using sequence contrastive pretraining @xcite . pretrained models with billions or more parameters also improve dr accuracy, though the benefit of scaling in dr is less than observed in other tasks @xcite . dr often serves as the first stage retrieval in many language systems. jointly training dr with later stage models can lead to better accuracy, with labels and signals from more sophisticated models in later stages, for example, question answering systems @xcite and reranking models @xcite . ## dense retrieval training analysis in this section, we first present the preliminaries of a standard dense retrieval setup and then investigate its training instability issues. ## teleportation negative sampling in this section, we present ance-tele, which introduces teleportation negatives to ance training. motivated by our analysis, ance-tele unions the training negatives along the time-axis of ance episodes, with the goal of smoothing training signal changes, improving negative group coverage, and, ultimately, reducing catastrophic forgetting. the construction of teleportation negatives of ance-tele is illustrated in figure 1 . specifically, the training episode i is defined as: ance-tele utilizes the same iterative training, but introduces two new components to standard ance negatives around q (eqn. 4), momentum negatives (eqn. 5) and lookahead negatives (eqn. 6): momentum negatives include training negatives from past episodes using a momentum queue (eqn. 5). they smooth out the training signal changes between past and current episodes. a standard technique to reduce catastrophic forgetting @xcite , including training signals from previous learning stages reminds the model to keep their learned knowledge. lookahead negatives predict the potential hard negatives in future training episodes using the neighbors of the positive document (eqn. 6). information from future is known to be beneficial in many other scenarios, for example, to probe the optimization space @xcite and to guide language generations @xcite . however, obtaining future information often requires extra costly operation steps, e.g., actually performing an extra ance episode to collect. instead, we propose to use the neighbors of d + to approximate the negatives that may appear in future episodes, i.e., "lookahead". intuitively, as the dense retrieval training is to pull closer q and d + in the representation space, a side effect is that the neighbors around d + are also pulled to the query. figure 7 shows that in ance training d + neighbors indeed contribute to a large fraction of new negatives introduced in future episodes, a handy feature for ance-tele to efficiently incorporate future learning signals. the sampling weights of momentum and lookahead negatives are controlled by hyperparameters (α, β), which we simply set as 0.5 without tuning. eliminating dependency on sparse retrieval. as shown in the last section, the sparse retrieval negatives are dropped after the warm up episode. ance-tele thus directly starts from itself: this removes the dependency of dense retrieval training on sparse retrieval. ance-tele introduces little computation overhead. the momentum negatives can be cached on disk and merged when constructing the training data for new episodes. the lookahead negatives need one extra ann retrieval operation per positive document, which is efficient @xcite . none of them increases gpu computations, which are the bottleneck in training. other than adding teleportation negatives, ance-tele keeps other system components intact and can be plugged into with most dense retrieval systems. ## experimental methodology we describe our general experimental settings in this section and leave more details in appendix. ## evaluation results this section first evaluates ance-tele and its ablations. then we analyze the influences and characteristics of teleportation negatives. ## conclusions we present ance-tele, a simple approach that effectively improves the stability of dense retrieval training. our investigation reveals the issues underlying the training instability: the catastrophic forgetting and negative swing behaviors. ance-tele resolves these issues by introducing teleportation negatives, which smooth out the learning process with momentum and lookahead negatives. ance-tele leads to strong empirical performance on web search and openqa with improved training stability, convergence speed, and reduced catastrophic forgetting. our analysis demonstrates the benefits of teleportation negatives and their behavior during training iterations. overall, ance-tele addresses an intrinsic challenge in dense re-trieval training with reduced engineering effort and minimum computation overhead. it can be used as a plug-in upgrade for the first stage retrieval of many language systems.