DSI++: U PDATING T RANSFORMER M EMORY WITH N EW D OCUMENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Differentiable Search Indices (DSIs) encode a corpus of documents in the parameters of a model and use the same model to map queries directly to relevant document identiﬁers.
Despite the solid performance of DSI models, successfully deploying them in scenarios where document corpora change with time is an open problem.
In this work, we introduce DSI++, a continual learning challenge for DSI with the goal of continuously indexing new documents while being able to answer queries related to both previously and newly indexed documents.
Across different model scales and document identiﬁer representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents.
We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning.
To mitigate these issues, we investigate two approaches.
The ﬁrst focuses on modifying the training dynamics.
Flatter minima implicitly alleviates forgetting, so we explicitly optimize for ﬂatter loss basins and show that the model stably memorizes more documents (+12%).
Next, we introduce a parametric memory to generate pseudo-queries for documents and supplement them during incremental indexing to prevent forgetting for the retrieval task.
Extensive experiments on a novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution mitigates the forgetting in DSI++ by a signiﬁcant margin and improves the average Hits@10 by +21 .
1% over competitive baselines for NQ.
I NTRODUCTION
Differentiable Search Indices (DSIs; Tay et al.
(2022)) represent a new modeling paradigm for information retrieval tasks using sequence-to-sequence learning.
Speciﬁcally, DSIs leverage Transformer memory (Vaswani et al., 2017) to encode all of the information in a corpus of documents and then use that memory to answer user queries directly, thereby simplifying the retrieval process.
DSIs achieve this functionality by jointly optimizing for indexing (or memorization) and retrieval tasks.
The indexing task requires learning a mapping from document content to its identiﬁer, typically represented by integers or short strings (document identiﬁers, abbreviated docids ).
Then, the retrieval task necessitates mapping user queries to relevant docids.
Apart from its simplicity and end-to-end differentiable nature, DSI signiﬁcantly outperforms state-of-the-art “retrieve-and-rank” methods, based on dual-encoders (Ni et al., 2022).
Despite the remarkable performance of DSI models, there remain open questions about their applicability in the practical setting of dynamic corpora.
Consider the realistic scenario wherein new documents are continually added to the indexed corpus.
Updating the index in dual-encoder based methods requires computing embeddings for new documents, followed by re-indexing all document embeddings (Karpukhin et al., 2020).
In contrast, index construction using a DSI involves training a Transformer model.
Therefore, the model must be re-trained from scratch every time the underlying corpus is updated, thus incurring prohibitively high computational cost in comparison to dualencoders.
In this work, we aim to address this issue by devising methods for effective incremental indexing using Transformer memory, without re-training the DSI model from scratch.
Continual (or incremental) learning (Thrun, 1995; Parisi et al., 2019) is a biologically-inspired machine learning paradigm that deals with continuous learning of new tasks by preserving past knowledge and using it to efﬁciently learn new concepts.
Based on this paradigm, we propose DSI++ (DSI
D
D D
D D
D 5 Batch of documents for continual indexing
I n d e x i n g a cc u r a c y
Eval batch D 0 D
D(a) Explicit forgetting during continual indexing    100 Percentage of examples
N u m b e r o f f
o r g e tt i n g e v e n t s
OptimizerAdafactor
[Figure]
+ new documents) , a continual learning challenge for DSI to incrementally index new documents while maintaining the ability to answer user queries related to both previously and newly indexed documents.
To enable DSI++, we introduce a novel benchmark constructed from the existing Natural Questions (Kwiatkowski et al., 2019) dataset, simulating continual addition of documents to the system.
To the best of our knowledge, there is no prior work studying incremental learning for DSI.
A naive solution for DSI++ is to continuously ﬁne-tune the model with an indexing objective over new documents.
However, Figure 1a shows that continual indexing of new documents leads to catastrophic forgetting of the previously memorized documents (more details in §2.1), a common phenomenon in connectionist networks wherein learning of the new concepts interferes with the previously acquired knowledge (McCloskey & Cohen, 1989; French, 1999).
Furthermore, when we investigate the learning dynamics of the DSI model during memorization (Figure 1b, we observe a signiﬁcant number of documents (approx.
88% ) experience forgetting events after they have been memorized.
Concretely, a forgetting event (Toneva et al., 2019) is deﬁned as when an individual document goes from being classiﬁed correctly to incorrectly over the course of learning.
Therefore, implicit forgetting during memorization and explicit forgetting from continual indexing of new documents are two key challenges that must be overcome to successfully implement a DSI++ system.
Mirzadeh et al.
(2020) show that geometrical properties of the minima play a role in forgetting.
In particular, models in ﬂatter minima tend to undergo less forgetting.
Further, Mehta et al.
(2021) showed that explicitly optimizing for ﬂatter loss basins using Sharpness-Aware Minimization (SAM;
Foret et al.
(2021)) reduces forgetting.
Building on these works, we show that ﬂatter minima induced by SAM reduce implicit forgetting during memorization, thereby leading to more stable memorization.
Next, to alleviate explicit forgetting from continual indexing of new documents, we investigate how to effectively index both old and new documents.
We propose leveraging a parametric memory to generate pseudo-queries for old and new documents and use them during continual indexing to mitigate retrieval task forgetting.
Our main contributions can be summarized as follows:
• We introduce DSI++, a continual learning challenge for Differentiable Search Indexes.
To enable DSI++ evaluations, we create a benchmark based on the existing Natural Questions (Kwiatkowski et al., 2019) dataset.
To understand the severity of the forgetting phenomenon across multiple scenarios, we analyze a suite of pre-trained models (T5-Base, T5-Large, T5-XL) and different document identiﬁer representations.
• We hypothesize and verify that the DSI model experiences forgetting events throughout memorization.
To alleviate these, we propose modifying training dynamics to promote ﬂatter minima using SAM and show that the model stably memorizes +12% documents.
• We propose generative memory-based experience rehearsal approach to alleviate explicit forgetting during continual indexing and improve the average Hits@10 by +21 .
1% over considered baselines.
DSI++: C ONTINUAL LEARNING CHALLENGE FOR DSI 2. P ROBLEM SETUP
We focus on a setup where we receive an initial corpus of documents, D
= { d 1 , · · · , d n } , and user queries corresponding to a subset of them, R
= { < q j , j >, ∀ j ∈ Y D } , where D ⊂ D 0 .
DSI paradigm involves two tasks: (i) memorization task where the goal is to learn an indexer f θ : X →
Y , a text-to-text model like T5 (Raffel et al., 2020) parameterized by θ ∈ R P , that takes document tokens ( x ∈ X ) as input and maps it to a document identiﬁer (docid) j ∈ Y , and (ii) retrieval task where the goal is to use the same indexer f θ to directly map a user query q to a relevant docid j ∈ Y .
Following Tay et al.
(2022), two different prompts are used to differentiate between these tasks.
Tay et al.
(2022) discuss several variants for representing docids – unstructured atomic and structured string docids, where each document is assigned a unique token and tokenized string, respectively.
Under the uniﬁed text-to-text format, both of the above tasks are cast as generation tasks, i.e., decoding one unique token (unstructured atomic) or decoding a tokenized string sequentially, one token at a time (naively/ semantically structured).
For DSI evaluation, we report indexing accuracy for memorization task and Hits@N ( N ∈ { 1 , 10 } ) metric for retrieval task.
Indexing accuracy and Hits@N are deﬁned as the proportion of documents correctly memorized and the proportion of correct documents ranked in the top N predictions, respectively.
To simulate a dynamic corpus scenario, we sequentially update the D 0 corpus with batches of documents D 1 , D 2 , ... where D 1 arrives ﬁrst followed by D 2 , and so on.
Moreover, with the new D i corpus, we do not assume any queries corresponding to documents.
Goal: Learn a DSI++ system that incrementally indexes D 1 , D 2 , · · · in f θ while being able to answer queries related to previously as well as additionally indexed documents.
2. B ENCHMARK FOR DSI++.
To enable research on DSI++, we introduce a new benchmark constructed from the Natural Questions dataset (NQ; Kwiatkowski et al.
The NQ dataset consists of Wikipedia articles and corresponding natural language questions.
Similar to Tay et al.
, we consider Wikipedia articles for memorization and the retrieval task as identifying the Wikipedia article that answers the given question.
We use the original NQ train split to construct train( 80% )/ validation( 20% ) splits and use NQ validation as a test split for our setup.
We randomly sample 50 K unique articles to constitute the initial D 0 corpus.
Next, we construct ﬁve corpora ( D 1 , · · · , D 5 ), each containing 10 K unique articles, to sequentially add them to the DSI model.
Corresponding to articles in each of these corpora, we ﬁlter queries from original NQ train/ validation splits to construct R traini , R vali , R testi ( ∀ i ∈ { 0 , · · · , 5 } ) splits.
Except for R 0 , one only requires R testi to evaluate for retrieval on previously and newly indexed articles.
Table 2 (in Appendix A.1) reports exact dataset statistics.
Similarly, we construct another benchmark from the MS MARCO dataset (Nguyen et al., 2016).
2. E VALUATION M ETRICS
In this subsection, we formally deﬁne metrics to summarize the model performance as we incrementally index new documents.
Let P n,o denote the performance (e.g., indexing accuracy) on corpus o after training on corpus n .
Following prior works (Lopez-Paz & Ranzato, 2017; Riemer et al., 2019), we compute the average performance ( A n ), forgetting ( F n ) and learning performance ( LA n ) metrics after indexing the corpus n .
F n (or backward transfer ) measures the inﬂuence of continual indexing the corpus n on the performance of all previously indexed documents o, (0 ≤ o < n ) .
LA n measures the learning capability when the model sees a new corpus n (or forward transfer ) and is deﬁned to be average over new corpora ( 1 , · · · , n ).
Say we incrementally index n th corpus, then A n , F n and LA n are deﬁned as follows:
[Equation]
D 0 D 1 D 2 D 3 D 4 D
I n d e x i n g a cc u r a c y Average performance ( A n )
D
D D
D D
Forgetting ( F n )
D D
D D
D 100 Learning performance ( LA n )
T5-BaseT5-LargeT5-XL
T5-Base(N)T5-Base(S)
D 0 D
[Figure]
D
[Figure]
D
[Figure]
H i t s @
D
D D
[Figure]
D
[Figure]
D
D D
[Figure]
2. C ASE STUDY : C ATASTROPHIC F ORGETTING AND F ORWARD T RANSFER
After introducing the DSI++ problem setup, benchmark, and evaluation metrics, we study the behavior of the DSI model as new documents are continuously added to the system.
Concretely, we are interested in investigating the following for continual training of the DSI model with indexing objective on new documents – (Q1)
How severe is the forgetting for the originally indexed documents?
, (Q2)
How does continual updating of the DSI model over a sequence of corpora affect the forgetting?
, (Q3)
How does the updated DSI model perform on newly indexed documents, especially the retrieval task?
, (Q4) How do different docid representation strategies affect forgetting?, and (Q5)
How does the DSI model scale affect forgetting?
Figure 2 visualizes results (averaged over 3 runs) on the validation split of DSI++ and help us convincingly answer these questions.
Forgetting (or negative backward transfer).
From Figure 2, we see that T5-Base model with atomic docid representation (blue line plots), undergo signiﬁcant forgetting and this trend holds across all DSI evaluation metrics - indexing accuracy, Hits@1, and Hits@10 (see 5 in Appendix
A.3).
For the originally indexed D 0 corpus, indexing accuracy and Hits@1 drop by approx.
25 and 20 points, respectively.
Further, as we continue indexing the sequence of D 1 , · · · , D 5 corpora, we see that forgetting becomes even more severe.
For example after continual indexing the D 5 corpus,
F 5 (forgetting) for indexing accuracy increases to 75 .
These results provide evidence to answer (Q1) & (Q2) that the DSI model undergoes severe forgetting under continual indexing of new documents.
Forward transfer.
To answer (Q3), we visualize the learning performance ( LA n ) for all DSI metrics for sequential indexing.
From Figure 2, we see LA n increases for indexing accuracy, suggesting that the DSI model is plastic enough to index new documents.
However, from Figure 2, we see a declining trend for Hits@1.
Due to the continuous indexing updates, the underlying DSI model drifts and becomes less effective for the retrieval task.
These ﬁndings hint at an approach that replays indexing and retrieval tasks during continual learning (hence our proposed method in Section 4).
Docid representations.
For studying (Q4), we consider unstructured atomic, naively(N) structured, and semantically(S) structured docid representations.
From Figure 2, we see that T5-Base(N) underperforms T5-Base by a signiﬁcant margin.
For example, average performance A 0 for Hits@ metric is approx.
30 and 39 for naive and atomic docids, respectively.
Furthermore, as the naively structured approach treats unstructured docids as tokenizable strings as opposed to dedicated unique tokens in the case of atomic docids, they are relatively more prone to interference from new docids (see F n subplot for indexing accuracy).
Imbuing semantic structure to the naive docid space helps to reduce forgetting however still underperforms unstructured atomic docids.
[Figure]
I n d e x i n g a cc u r a c y
OptimizerAdafactorSAM (
[Figure]
Indexing accuracy during memorization
[Figure] [Figure]
Percentage
[Figure]
examples
N u m b e r o f f
o r g e tt i n g e v e n t s
OptimizerAdafactorSAM
[Figure]
Model scale.
As atomic docids are superior to naive docids, we only consider atomic docids for answering (Q5).
From Figure 2, we observe that larger models outperform their smaller counterparts in terms of the average performance A n and the learning performance LA n (T5-XL > T5-Large > T5Base).
However, empirically we report that forgetting F n is severe across all model scales, without any clear best performer, and therefore, we focus on T5-Base for the rest of our experimentation.
I MPLICIT F ORGETTING DURING M EMORIZATION : SAM
Memorization (or indexing) is a primary task in the DSI paradigm (Tay et al., 2022) where the goal is to learn a neural corpus indexer that takes document content as input and maps it to a document identiﬁer (docid).
Under unstructured atomic docid representation strategy, each docid is assigned a unique token / class label.
Now given the large number of documents in the corpus (even more than a million), memorization constitutes an instance of challenging extreme classiﬁcation setting (Bengio et al., 2019).
Furthermore, for every class, we have only one labeled example (i.e., document and its identiﬁer), making this task setup rare.
Motivated by this largely unexplored setup, we investigate the learning dynamics for the memorization task over the course of training.
Forgetting events.
In Figure 3a, we visualize the indexing accuracy for T5-Base model, optimized with Adafactor (Shazeer & Stern, 2018), and note that the model performance ﬂuctuates a lot over the course of training, thereby suggesting unstable memorization.
We hypothesize that the model continuously undergoes the forgetting phenomenon wherein subsequent mini-batch updates interfere with the previously memorized documents.
To differentiate this phenomenon from forgetting due to adding new documents, we refer to earlier one as implicit forgetting and the latter as explicit forgetting .
To quantify instability during memorization, we compute forgetting event (Toneva et al., 2019) statistics.
Forgetting event is deﬁned when an individual document goes from being classiﬁed correctly to incorrectly over the course of memorization.
In Figure 3b, we plot the cumulative histogram of forgetting events and see that almost 88% of the documents undergo forgetting at least once, thus, validating our hypothesis about implicit forgetting.
Flatter minima and forgetting.
Mirzadeh et al.
(2020) shows that during sequential learning of tasks, geometric properties of the minima for each task affect forgetting.
Speciﬁcally, Mirzadeh et al.
(2020) derives an upper bound for forgetting in terms of the maximum eigenvalue λ maxof the Hessian for task loss at minima.
Therefore, by lowering λ max(or alternatively promoting ﬂatter minima), one can mitigate forgetting of previous tasks.
Further, Mehta et al.
(2021) shows that pre-trained initialization implicitly alleviate forgetting as they prefer ﬂatter minima and explicitly optimizing for the ﬂatter loss basins using Sharpness-Aware Minimization (Foret et al., 2021)
(SAM) further lessens forgetting.
Based upon these observations, we hypothesize that modifying the training dynamics of memorization task using SAM should alleviate implicit forgetting.
Sharpness-Aware Minimization.
For the loss function f , SAM seeks to ﬁnd the parameters w that lie in the neighborhood with uniformly low loss regions by optimizing the following minimax objective: min w max || (cid:15)
|| 2 ≤ ρ f ( w + (cid:15) ) , where the maximization region is deﬁned to be a (cid:96) p ball with radius ρ for p = 2 .
Foret et al.
(2021) estimates the gradient of the inner maximization by employing ﬁrst-order approximation as follows: ∇ w max || (cid:15)
|| 2 ≤ ρ f ( w + (cid:15) )
≈ ∇ w f ( w ) (cid:12)(cid:12) w + ˆ (cid:15) ( w ) , where ˆ (cid:15) ( w ) =
ρ ∇ w f ( w ) / ||∇ w f ( w ) ||
For complete details about this derivation, we defer readers to (Foret et al., 2021).
Building on this work, Bahri et al.
(2022) demonstrates successful applicability of SAM for language model generalization, especially in pre-trained T5 models.
We mainly follow (Bahri et al., 2022) to set our hyper-parameters: ρ = 0 .
15 , batch size= 32 for inner maximization step.
SAM alleviates Implicit forgetting.
We investigate the applicability of SAM for alleviating implicit forgetting phenomenon.
Concretely, we use pre-trained T5-Base model to memorize D 0 corpus, containing 50 K unique documents.
We compare the performance of the SAM optimizer with the vanilla Adafactor optimizer.
In Figure 3a, we see that SAM outperforms Adafactor in terms of the overall indexing accuracy.
Further, we note that SAM undergoes less severe ﬂuctuations during the course of training, thus, hinting at lesser forgetting.
To bolster this claim, in Figure 3b, we see that
SAM has signiﬁcant more percentage of documents corresponding to lower cumulative number of forgetting events.
For example, SAM stably (with zero forgetting events) memorizes +12% more documents compared to Adafactor.
We also note that SAM ( 35 . 9 ± 2 . 2 ) outperforms Adafactor ( 32 . 5 ± 6 . 4 ) when evaluated on the retrieval task (Hits@1) corresponding to D 0 .
Therefore, we set
SAM to be our default optimizer for rest of the experiments.
E XPLICIT F ORGETTING : G ENERATIVE M EMORY
DSI paradigm consists of two tasks – memorization and retrieval.
In the previous section, we showcase that SAM alleviates implicit forgetting by stably memorizing documents.
In this section, we focus on the forgetting phenomenon that arises from the continual indexing of new documents, speciﬁcally in the context of the retrieval task.
Through our systematic study (in Section 2.4), we show that irrespective of the model scale and docid representations, DSI models undergo severe forgetting.
Moreover, we observe that the learning performance LA n keeps declining for the retrieval task (see Figures 2 and 5 for Hits@1 and Hits@10, respectively).
This observation suggests that as we continuously update the DSI model with the indexing objective, the model starts to forget the retrieval task.
In DSI, both memorization and retrieval tasks return docid for the given input, so by setup, we can assume access to the contents of the previous documents and continue indexing both old and new documents with the hope to reduce forgetting of the retrieval task.
However, in Figure 4, we see that the model still undergoes forgetting (more discussion in Section 5.3).
Episodic memory.
According to the Complementary Learning Systems (CLS) (McClelland et al., 1995) theory, humans rely on an episodic memory – a module that stores past experiences to rehearse/ revisit them and thereby retain previously learned knowledge.
Based on this motivation, memory-based approaches (Sodhani et al., 2022) for continual learning use a subset of previous task data to regularize the future task learning while minimizing forgetting.
Experience Replay (ER) (Chaudhry et al., 2019) is one such approach that samples previous task data (from episodic memory) to co-train with the current task.
Based upon this line of work, one approach for DSI++ is to retain ground-truth queries for the retrieval task in an episodic memory and use them to co-train with incremental indexing task.
However, in DSI++, we do not have access to ground-truth queries for an incoming stream of new documents.
Even if one retains queries for the initial D 0 corpus, we show in Table 1 that such a method suffers from forward transfer to newly indexed documents.
Generative memory.
Recent years have seen signiﬁcant progress in the capabilities of the generative language models (Raffel et al., 2020; Brown et al., 2020).
Motivated by the success of these models and in-applicability of the episodic memory for DSI++, we pose a questions – instead of retaining the ground-truth queries, can we learn a parametric model to generate such queries given a document?
Concretely, we propose to train a query generator model to sample queries for previously seen documents, and supplement them during incremental indexing.
Since we use the generator model to sample queries for sparse experience replay, hence our proposed method – generative memory .
Moreover, generative memory is also used to generate pseudo-queries for incoming batch of new documents, thus, enabling semi-supervised learning of the retrieval task.
[Table]
1 ): continue ﬁne-tuning with indexing task on D 1 , cl( U 1 ): continue ﬁne-tuning on the updated corpus U 1 , cl( U 1 )+epsmem( D ): continual indexing of U 1 along with ER of queries for D , cl( U 1 )+genmem( D ): continual indexing of U 1 along with ER of pseudo-queries for D .
We observe that continual indexing on the updated corpus cl( U 1 ) reduces forgetting in comparison with just indexing new corpus cl( D 1 ).
Next ER with either D 0 or D 1 hurts forward transfer or forgetting, respectively.
Our proposed approach of augmenting pseudo-queries for all documents along with continual indexing, cl( U 1 )+genmem( U 1 ), alleviates forgetting and improves forward transfer.
E XPERIMENTATION 5. I MPLEMENTATION DETAILS
We utilize the pre-trained T5-Base (Raffel et al., 2020) model to initialize all models and randomly initialize the additional parameters for atomic docid tokens.
While indexing D 0 corpus, we train all the models for a maximum of 1 M steps with a warmup of 100 K steps.
During continual indexing of other corpora, we train for a maximum of 100 K steps with a warmup of 100 steps.
For the rest of hyper-parameters, we follow Tay et al.
(2022) – set a learning rate to 0 .
001 , batch size to 128 , and input sequence length to 32 .
We evaluate models after every 5 K steps and retain the checkpoint yielding the best performance.
For the initial training with D 0 corpus, we co-train on indexing and retrieval tasks, therefore we use the average of all DSI metrics (indexing accuracy,
Hits@1, and Hits@10) for model selection.
For the continual learning experiments, we have access to only indexing accuracy for all involved corpora and so we use it for model selection.
For training generative memory, we use retrieval dataset R 0 (corresponding to D 0 corpus).
We set the maximum sequence length for document contents to 1024 , target length for generated queries to 32 , batch size to 128 , train for a maximum of 100 K steps, and use BLUE for model selection.
We use beam decoding to generate pseudo-queries.
We tune the learning rate amongst { 0 . 001 , 0 . 0005 } and linear warmup amongst { 1 K, 10 K } .
For all our experiments, we use T5X
(Roberts et al., 2022) framework along with 4 - 8 TPUv4 chips for training the models.
5. M ETHODS
We compare our proposed generative memory with following methods:
• continual indexing, cl( D n ).
The DSI model is sequentially ﬁne-tuned with the indexing objective on the incoming corpus of documents D n .
• continual indexing with all seen documents, cl( U n ).
The DSI model is continuously ﬁne-tuned with the indexing objective on the updated corpus U n ( = (cid:83) ni
=
D i ).
Also, we sample documents from old ( (cid:83) n − 1 i =
D i ) and new ( D n ) corpora in equal proportion.
• experience replay using generative memory, genmem( D n ).
In this method, the proposed generative memory model is used to generate pseudo-queries corresponding to the corpus D n .
Next, these pseudo-queries are used for experience replay of the retrieval task samples.
• experience replay using episodic memory, epsmem( D n ).
In this method, ground-truth queries corresponding to the D thn corpus are used for experience replay of the retrieval task.
[Figure]
5. R ESULTS
In this section we revisit some of the questions, (Q1)-(Q3), raised in our case study (see Section 2.4) to investigate effectiveness of our proposed generative memory-based approach.
Towards answer these questions, in Table 1, we report the performance of the DSI model on D 0 (to study forgetting phenomenon) and D 1 corpora (to answer forward transfer question) after continual indexing on D 1 .
In Figure 4, we report overall performance across DSI metrics as we continuously update the model with the sequence of ﬁve corpora ( D 1 , · · · , D 5 ).
See Table 3 for results on MS MARCO dataset.
Does generative memory alleviate forgetting of old documents?
In Table 1, we report Hits@ to be 35 .
9 for the model after training on D 0 .
We see that continually indexing both D 0 and Dcorpora (cl( U 1 ) - 28 . 9 ), signiﬁcantly reduce forgetting the retrieval task (Hits@1) over just indexing the new corpora D 1 (cl( D 1 ) - 19 . 2 ).
Next, we look at the performance of the ER approaches when augmented with the continual indexing of all documents.
We see that both episodic memory (cl( U 1 )+epsmem( D 0 ) - 22 . 9 ), and generative memory (cl( U 1 )+genmem( D 0 ) - 26.0) reduce forgetting compared to cl( D 1 ) when we replay (pseudo-)queries corresponding to D 0 corpus.
Moreover, generative memory outperforms episodic memory, without retaining original queries.
Although from Table 1, we see generative memory underperforms cl( U 1 ), from Figures 4 and 6, we see that generative memory outperforms cl( U 5 ) both in terms of average performance A n and forgetting F n over ﬁve sequential updates.
These results convincingly show that the ER with generative memory signiﬁcantly alleviate forgetting the retrieval task compared to considered baselines.
Does generative memory enable forward transfer to new documents?
One of the goals of DSI++ is to enable answering queries related to newly indexed documents.
Towards this goal, in Table 1, we look at the retrieval task performance (Hits@1 metric) for D 1 after incrementally indexing D 1 .
To compare different methods, we consider a baseline in the form of ER with ground-truth queries for
D 1 (cl( U 1 )+epsmem( D 1 ) - 48.3).
We see that without any ﬁne-tuning on the retrieval task for D 1 , incremental learning with indexing objective shows impressive forward transfer (or zero-shot gains, cl( D 1 ) - 31.7 and cl( U 1 ) - 34.0).
Moreover, ER with generative memory outperforms supervised baseline (cl( U 1 )+genmem( D 1 ) - 57.7).
However, we notice that replaying queries corresponding to either D 0 or D 1 hurt forward transfer to D 1 (cl( U 1 )+genmem( D 0 ) - 8.6) or amplify forgetting of
D 0 (cl( U 1 )+genmem( D 1 ) - 7.0), respectively.
These results suggest that memory module should include (pseudo-)queries corresponding to both old and new documents.
From Figure 4, we see that continual indexing method cl( U n ) has a downward trend for LA n (Hits@10), therefore, eventually forgetting the retrieval task.
On the other hand, ER with generative memory is relatively constant, providing evidence against forgetting.
In summary, we show that ER with the generative memory improves the overall performance for the retrieval task, reducing forgetting of previously indexed documents and enabling forward transfer to newly indexed documents.
Investigating sparsity of experience replay (ER) on forgetting.
ER with generative memory cotrains the indexing and pseudo-labeled retrieval tasks.
Tay et al.
(2022) introduces a mixing ratio r to deﬁne the ratio of indexing to retrieval samples.
The mixing ratio is inversely related to the sparsity of ER, i.e., higher r (more indexing samples) corresponds to sparse updates from pseudolabeled retrieval samples.
Following (Tay et al., 2022), we consider r = { 2 , 32 } for our analysis.
From Figure 4, we see that r = 32 (sparse replay) slightly outperforms r = 2 in terms of average performance, forgetting and learning accuracy.
These results suggest that even sparse regularization updates from ER positively inﬂuence the backward and forward transfer in DSI++.
Analyzing index construction time for DSI++.
Index construction using DSI involves training a
Transformer model.
DSI++ setup enables incremental updating of the indexer.
We require 350 K training steps to index D 0 corpus of 50 K documents.
If we index an additional D 1 to D 5 corpora ( 10 K each) by re-training the DSI model every time, the total number of steps would be around 1 .
75 M.
On the other hand, our proposed approach requires just above 300 K additional updates to incrementally index all ﬁve corpora, almost 6 times fewer updates.
R ELATED W ORK
We review relevant prior works along two dimensions – application setups related to DSI++, and approaches to alleviate forgetting during continual learning.
Petroni et al.
(2019) views pre-trained language models as knowledge bases.
Based on this Zhu et al.
(2020) introduces an experimentation setup where task is to update facts stored within the pre-trained models.
Next, De Cao et al.
(2021) introduces, KnowledgeEditor, a hyper-network based method to efﬁciently update the facts.
Although interesting line of works around fact updation, it is challenging to know whether pre-trained models have actually learned the fact or our probing mechanism is unable to extract the relevant facts.
On the other hand, we explicitly focus on the memorization task in DSI++.
This helps us to answer questions related to catastrophic forgetting more convincingly rather than bounded by the mechanism how we probe these models.
For a ﬁxed capacity model – (1) Regularization-based approaches constrain the model updates by adding a penalty term to the original objective.
However, these approaches are too stable and prevent learning new tasks, (2) Optimization-based approaches encode the necessary inductive biases required to enable continual learning by modifying the training dynamics.
Flatter minima are shown to alleviate forgetting (Mirzadeh et al., 2020; Mehta et al., 2021).
Our work extends on this line of research and showcases applicability of SAM for minimizing implicit forgetting during memorization (see Section 3).
(3) Memory-based (aka data-based regularization) approaches constrain the parameter updates based upon the previous task examples sampled from memory.
Sparse experience replay using episodic memory (Chaudhry et al., 2019) is a prominent approach and in Section 4, we discuss limitations of it for DSI++.
Shin et al.
(2017); Sun et al.
(2020) learns a parametric model to reconstruct the examples from previous tasks.
However, (Sun et al., 2020) shows that such approach still underperforms episodic memory.
In our work, we do not generate example pairs ( x, y ) but rather generate pseudo-queries ( y ), similar to concurrent works (Zhuang et al., 2022; Bonifacio et al., 2022).
We show that our approach outperforms episodic memory.
C ONCLUSION
DSI++ presents a novel direction for exploration of DSI models to solve one of the critical requirements for them to be usable in most production setups where new documents are added to the corpus continuously.
Though DSI models are prone to catastrophic forgetting while we index new documents, we have shown through extensive experiments that our two proposed solutions to optimize for the ﬂatter loss basins using SAM and using generative memory alleviate forgetting to a signiﬁcant extent.
With this work, we lay down foundations for further research in this space, so that the strategies here do not just beneﬁt DSI, but the continual learning community in general.
E THICS S TATEMENT
Training large models is expensive, and has a detrimental impact on the environment.
Continual learning on top of existing models is cheaper and better compared to retraining from scratch since it requires a much smaller number of steps.
With DSI++, we aim to reduce the need to retrain DSI models from scratch whenever a new set of documents is added to the corpus thereby making it cheaper and better for the environment.
R EPRODUCIBILITY S TATEMENT
We introduce a novel benchmark constructed from the existing and publicly available Natural Questions (NQ) (Kwiatkowski et al., 2019) and MS MARCO (Nguyen et al., 2016) datasets.
In our benchmark, we split the NQ and MS MARCO datasets into different subsets to enable development of continual learning systems.
We plan to release these splits so that they can act as a standard benchmark for future works in this space.
Further, in Section 3 and Section 5.1, we detail all hyperparameters to enable reproducibility of all our experiments.
We also plan to release our code on publication
A A PPENDIX A. D
[Table]
A. MS MARCO
In Table 3, we report Hits@1 to be 78 . 0 for the DSI model after training on D 0 (constructed from
MS MARCO dataset).
We see that continually indexing both D 0 and D 1 corpora (cl( U 1 ) - 75 .and
cl( U 1 )+genmem( U 1 ) - 73 . 0 ), signiﬁcantly reduce forgetting the retrieval task (Hits@1) over just indexing the new corpora D 1 (cl( D 1 ) - 65 . 5 ).
Next, we look at the retrieval task performance (Hits@1 metric) for D 1 after incrementally indexing D 1 .
We see that without any ﬁne-tuning on the retrieval task for D 1 , incremental learning with indexing objective shows impressive forward transfer (or zero-shot gains, cl( D 1 ) - 47.6 and cl( U 1 ) - 43.6).
Moreover, ER with generative memory, cl( U 1 )+genmem( U 1 ) - 80.6, performs far superior to just incremental indexing objective.
In summary, we show that ER with the generative memory improves the overall performance for the retrieval task, reducing forgetting of previously indexed documents and enabling forward transfer to newly indexed documents.
We show that our results hold across two datasets – Natural Questions and MS MARCO, thus, showcasing the generalizability of our approach.
[Table]
A. A DDITIONAL R ESULTS
[Figure] [Figure]
F n ) but also improves average Hits@1 ( A n ) by +17 .
3% over considered baselines