From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Overall Response
comment: Hi reviewers,

We thank you all for your detailed comments and valuable suggestions! Given we are close to the end of the final discussion period and have not heard back from anyone of you, we would appreciate it if you could review our responses once! We hope that our revisions and responses address all of your concerns. 

We are glad that the reviewers find: (i) our paper to be well-written, coherent, comprehensive, and clear in terms of the presentation (reviewers mfHG, YCgn, uo3e), (ii) our work is the first to study the vital problem of updating DSI and appreciates our efforts in this direction (reviewers mfHG, YCgn, uo3e), (iii) our experiments are comprehensive with interesting results (reviewers mfHG, uo3e), and (iv) our introduced benchmarks are valuable to the community (reviewer mfHG).

We have provided a detailed response to each reviewer and uploaded the revised manuscript. Here we summarize the key findings from additional experiments:

1. We introduce **one more benchmark for DSI++**, constructed from a publicly available **MS Marco passage retrieval dataset**. We conduct extensive experiments on this benchmark and [show similar findings as the NQ dataset, thereby, strengthening our claims.](https://openreview.net/forum?id=XkwkFYPT6t&noteId=9-QYZK-ijY)

2. To study the scalability of our approach to a large number of documents, we conduct additional experiments with **8.9M MS Marco passages** and [showcase improvements both in terms of alleviating forgetting of the previously indexed 8M passages and forward transfer to the newly indexed approx. 900K passages.](https://openreview.net/forum?id=XkwkFYPT6t&noteId=9-QYZK-ijY)

3. We also conduct experiments with **semantically structured docids** and [showcase that imbuing the docid space with a semantic structure alleviates forgetting compared to an arbitrary/ naive structure; however, still underperforms unstructured atomic docids.](https://openreview.net/forum?id=XkwkFYPT6t&noteId=9-QYZK-ijY)

We are happy to answer any further questions.

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Response to Reviewer mfHG (Part 1/1)
comment: We are glad that the reviewer finds (i) our work addresses a **vital aspect** of the original DSI paradigm, i.e., incremental adding new documents to the DSI model, (ii) the **paper is clear** in terms of the presentation, especially details about data, metrics, training, and evaluation, (iii) our results are **interesting**, and (iv) our introduced benchmarks are **valuable to the community**. 

**[Relevant baselines]**

The reviewer points out that our work lacks comparison with relevant prior work, specifically those belonging to the “retrieval augmented generation” family of approaches [Khandelwal et. al (2020); Guu et. al (2020); Borgeaud et. al (2021); Izacard et. al (2021)]. These approaches retrieve auxiliary passages/documents to enhance pre-trained language models. Basically, they alter test-time predictions of the generative models by augmenting their input with relevant passages retrieved from external retrievable memory. Moreover, by using the external retrievable memory, one is explicitly disabling the updates to the employed pre-trained (and retrieval) model. **We argue that such approaches do not faithfully assess the fundamental challenge of learning continually, specifically catastrophic forgetting.**

On the other hand, our work focuses on the **recently introduced DSI paradigm**, where information retrieval is encapsulated into a complete end-to-end learnable paradigm, i.e., using generation to replace the retrieval process (as opposed to using retrieval to augment a generation process). The follow-up work based on **the DSI paradigm recently bagged an outstanding paper at NeurIPS 2022**, speaking for the **increased attention from the community towards this new paradigm**. In the DSI paradigm, information in the document corpus is encoded into the model parameters. Therefore, any updates to the underlying corpus necessitate updating the model parameters hence, undergoing severe forgetting. **Our work tackles a more challenging setup for studying the forgetting phenomenon in detail. However, retrieval-augmented generation-based methods do not analyze the forgetting phenomenon at all, only looking at overall performance metrics.** 

We agree that continual learning is broader than catastrophic forgetting, however, in this work we decide to study the forgetting phenomenon in detail on one of the challenging setups, if not the most challenging. Therefore, **we respectfully and firmly disagree with the reviewer that the fact we do not compare with retrieval augmented generation methods is a ground for a clear rejection of our work.**

**[References]**

[Guu et. al (2020)] Guu, Kelvin, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. "Retrieval augmented language model pre-training." In International Conference on Machine Learning, pp. 3929-3938. PMLR, 2020.

[Izacard et. al (2021)] Izacard, Gautier, and Édouard Grave. "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering." In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880. 2021.

[Khandelwal et. al (2020)] Khandelwal, Urvashi, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. "Generalization through Memorization: Nearest Neighbor Language Models." In International Conference on Learning Representations. 2020.

[Borgeaud et. al (2021)] Borgeaud, Sebastian, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche et al. "Improving language models by retrieving from trillions of tokens." arXiv preprint arXiv:2112.04426 (2021).

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Response to Reviewer uo3e (Part 1/2)
comment: We thank the reviewer for their feedback and suggestions. We are glad that the reviewer acknowledges that: (i) our work is the **first to study** the problem of updating DSI, (ii) our experiments are **comprehensive and convincing**, and (iii) the paper is **well-organized and has a good writing style**.

We have detailed responses to the reviewer’s comments and look forward to hearing from the reviewer.

**[Document Identifier representation - Unstructured Atomic vs. Naively Structured vs. Semantically Structured Docids]**

Based upon the reviewer's suggestion, **we conduct an additional set of experiments with Semantically Structured Docids**. Following the original work [Tay et al. (2022)], we generate semantically structured docids for the NQ Wikipedia articles and use them for our constructed DSI++ NQ benchmark. In Figure 2 (page 4), we visualize our results with different docids - unstructured atomic vs. naively structured vs. semantically structured. We observe that naively structured docids, T5-Base(N), underperform unstructured atomic docids, T5-Base, across all metrics - indexing accuracy, Hits@1, (see Figure 5 in Appendix A.3 for Hits@10 results). **Imbuing the docid space with a semantic (S) structure alleviates the forgetting compared to an arbitrary/ naive (N) structure; however, still underperforms unstructured atomic docids.** 

**[Additional datasets]**

As suggested by the reviewer, **we introduce one more benchmark for DSI++, constructed from a publicly available MS Marco [Nguyen et al. (2016)] passage retrieval dataset**. Similar to the benchmark created from the NQ dataset, we randomly sample 50K unique passages to constitute the initial $D_0$ corpus, and five more corpora ($D_1, \cdots, D_5$), each with 10K passages (see Section 2.2 for more details). **In Table 3 (page 13), we report our results on the MS Marco benchmark for DSI++**. 

[Further, we investigate the effectiveness of generative memory in mitigating forgetting when continuously indexing MS Marco corpus $D_0 \rightarrow D_1 \cdots \rightarrow D_5$ (see attached figure)](https://drive.google.com/file/d/197u66gvommmdpOwOFxizkrfkwzna9VBT/view?usp=sharing). 

We observe that continual indexing of old and new documents cl($U_n$) helps to alleviate forgetting of older documents when evaluated on the MS Marco retrieval task. However, average Hits@1 ($A_n$) still undergoes 35 points drop after sequential updates ($D_0 \rightarrow D_1 \cdots \rightarrow D_5$). Generative memory enables sparse replaying of pseudo-queries for old and new documents. **We show that augmenting the generative memory during continual indexing of MS Marco passages reduces the forgetting ($F_n$) and improves average Hits@1 ($A_n$) by +25% over considered baselines.**

**[Scale of the document corpus - 8.9M passages]** 

As suggested by the reviewer, we also conduct experiments with a **full MS Marco** [Nguyen et al. (2016)] passage retrieval dataset containing **approx. 8.9M passages**. We construct two corpora – $|D_0| = 8M$ and $|D_1| = 841,823$ passages. We train the DSI model (unstructured atomic docid representations) using $D_0$ passages and incremental add $D_1$ passages. We report our results in the table below. **Our proposed generative memory-based approach i.e., augmenting pseudo-queries for all documents along with continual indexing, cl($U_1$)+genmem($U_1$), alleviates forgetting (on $D_0$ corpus) and improves forward transfer (on $D_1$ corpus) compared to continual fine-tuning with indexing task on $D_1$.**

| Added corpus | Method | Eval corpus = D0 (forgetting) | Eval corpus = D_0 (forgetting) | Eval corpus = D0 (forgetting) | Eval corpus = D1 (Forward transfer) | Eval corpus = D1 (Forward transfer) | Eval corpus = D1 (Forward transfer) |
| --- | --- | --- | --- | --- | --- | --- | --- |
|                  	|                   	| Index acc.                                    	| Hits@1           	| Hits@10          	| Index acc.                             	| Hits@1           	| Hits@10          	|
| D0               	| -                 	|                                         99.41 	|             16.3 	|             46.8 	|                                      - 	|                - 	|                - 	|
| D1               	| cl(D1)            	|                                      8.75e-05 	|              0.1 	|              0.6 	|                                   97.9 	|             18.2 	|             40.5 	|
| D1               	| cl(U1)+genmem(U1) 	|                                          20.4 	|              7.3 	|             31.3 	|                                   86.6 	|             31.6 	|             65.8 	|

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Response to Reviewer uo3e (Part 2/2)
comment: **[Full dataset baseline]**

[In the attached figure, we compare the performance of the generative memory-based approach (green and red) with the model trained from scratch on the full dataset (purple color) when evaluated on the NQ dataset](https://drive.google.com/file/d/1CvR3No3MtlCO4QOZunIyY2XrMMTjfr8M/view?usp=sharing). We see that the generative memory performs superior to the model trained from scratch across all three metrics – indexing accuracy, Hits@1, and Hits@10. 

Moreover, we require 350K training steps to index $D_0$ corpus of 50K documents. If we index an additional $D_1, \cdots, D_5$ corpora (10K each) by re-training the DSI model from scratch using the full dataset, the total number of steps would be around 1.75M. On the other hand, our proposed approach requires just above 300K additional updates to incrementally index all five corpora, **almost 6 times fewer updates**.

**In summary, the generative memory-based approach is superior to training the DSI model from scratch using the full dataset, both in terms of performance and computational requirements.**

**[Additional comments]**

1. We have updated our related work (see Section 6, page 9) to cite the concurrent work [Zhuang et al. (2022)] pointed out by the reviewer. However, our work investigates the problem of continually updating the DSI model using generative memory. On the contrary, the concurrent work [Zhuang et al. (2022)] primarily focuses on resolving the distribution discrepancies between the DSI indexing and retrieval tasks using data augmentation (generating queries for the documents) and is orthogonal to our work.

2. [We would like to point out the reviewer to our list of contributions again](https://openreview.net/forum?id=XkwkFYPT6t&noteId=UN_-3UHwrJ). We would request the reviewer to reconsider their technical and empirical novelty scores.

3. Regarding the reviewer’s concern about the scale of the document corpus – we would like to argue that the original DSI paper [Tay et al. (2022)] and the concurrent work [Zhuang et al. (2022)] evaluate their DSI models on the similar size (10-320K) of the corpus as our experimentation (50-100K). Nevertheless, we are currently experimenting with the full MS Marco dataset with 8.9M passages and will update the reviewers with our results.

4. In section 3, when we mention “explicitly adding documents”, we are referring to the scenario where we get new documents that we want our DSI model to index them. We are explicitly simulating the realistic data distribution shift in terms of the new documents. 

Please let us know if our answers above address your concerns. We are happy to answer any follow-up questions!


**References**

[Tay et al. (2022)] Tay, Yi, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin et al. "Transformer memory as a differentiable search index." arXiv preprint arXiv:2202.06991 (2022).

[Zhuang et al. (2022)] Zhuang, Shengyao, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. "Bridging the gap between indexing and retrieval for differentiable search index with query generation." arXiv preprint arXiv:2206.10128 (2022).

[Nguyen et al. (2016)] Nguyen, Tri, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. "MS MARCO: A human generated machine reading comprehension dataset." In CoCo@ NIPs. 2016.

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Response to Reviewer YCgn (Part 1/2)
comment: We thank the reviewer for their time and remarks on our work. We are pleased that the reviewer finds the paper **well-written, coherent, and comprehensive and appreciates our efforts in continually updating DSI**.

**[Novelty]**

We relist our contributions:

1. We **propose DSI++**, a continual learning challenge for the recently proposed Differentiable Search Indices (DSI) paradigm. To enable research progress on DSI++, we introduce **two benchmarks based on the existing Natural Questions and MS Marco datasets**. To understand the severity of the catastrophic forgetting, we **systematically study different design choices** like – document identifier representation strategies (unstructured atomic, naively structured, and semantically structured) and the pre-trained model size (T5-Base, T5-Large, T5-XL).


2. We are the **first to show that the DSI model undergoes a forgetting phenomenon during memorization**, leading to unstable learning dynamics. Mehta et al. (2021) show that explicitly optimizing for flatter loss basins using Sharpness-Aware Minimization (SAM) leads to lesser forgetting, especially in task-incremental learning settings where data undergoes a clear distributional shift. We extend this work to the new DSI paradigm and **convincingly demonstrate that SAM helps with the stable memorization of documents**. Our results **generalize the earlier findings even to the settings where data does not undergo a clear distributional shift** (i.e., memorization task). Although SAM helps to stably memorize +12% of documents, there is still a lot of room for improvement, and **our work invites more future work in this direction**.


3. We **propose a generative memory-based approach to alleviate forgetting during continual indexing of the new documents**. Note that in DSI++, we assume no access to ground truth queries for the older documents. Furthermore, we do not get to see queries for the new documents. Therefore, **we use generative memory to generate pseudo-queries for already indexed (older) documents as well as an incoming batch of new documents, thus, enabling us to leverage unlabeled data (in the form of new documents) for continual learning**. On the other hand, **Sun et al. (2020) assume that the incoming data are fully labeled, which is not applicable in DSI++** (we do not get to see queries for the new documents). Next, they learn a parametric model to reconstruct the examples for all of the seen tasks. However, such an approach strictly underperforms episodic memory which retains a subset of seen examples. In our work, we do not generate example pairs (document, queries) but rather generate pseudo-queries, and we show that our approach outperforms episodic memory that retains ground truth queries.


We believe that the above list of contributions would make the paper’s novelty clearer to the reviewer.


**[Experimental Setup]**

In DSI++, **we have access to the incoming stream of new documents**. However, **we do not have queries** corresponding to this stream of new documents. 

Our problem setup (see Section 2.1) does mention – "To simulate a dynamic corpus scenario, we sequentially update the D0 corpus with batches of documents D1, D2, ... where D1 arrives first followed by D2, and so on. Moreover, with the new Di corpus, we do not assume any additional metadata such as queries corresponding to documents."

From: ICLR.cc/2023/Conference/Paper5140/Authors
title: Response to Reviewer YCgn (Part 2/2)
comment: **[Additional baselines]**

The reviewer suggests additional baselines for DSI++, specifically from the parameter isolation family of methods [De Lange et al. (2021)]. Parameter isolation-based methods assign different **dedicated subsets of the model parameters to each task to prevent forgetting**. While learning a new task, these methods either **freeze a subset of the parameters corresponding to older tasks** or **dynamically add new parameters per new task.** At the prediction time, **these methods typically require task identity** to activate the corresponding subset of parameters for inference. 

In the DSI paradigm, we are given user queries at the inference time, and the goal is to predict relevant document identifiers. Now during incremental indexing, if we consider every new document corpus $d_i$ as a new task $t_i$ then **a typical parameter isolation-based approach would require corpus identity for every user query at the test-time, defeating the whole purpose of the DSI paradigm**. Due to this, the **parameter isolation-based approaches in their current form are rendered less useful for DSI++**. Nevertheless, we believe that by masking the weights for the already indexed corpus, one is explicitly disabling the updates to the underlying DSI model; therefore, parameter isolation-based methods would be robust to forgetting. We believe, however, that adapting these methods for DSI++ is out of scope for this paper, and we would not be able to do both this topic and our current work justice in the limited space available. 

**[Additional datasets]**

To showcase that our proposed approach generalizes to other datasets, **we introduce one more benchmark for DSI++**, constructed from a publicly available passage retrieval dataset, namely **MS Marco** [Nguyen et al. (2016)]. The full MS Marco dataset has approx. 500K passage-query training pairs and 6980 validation pairs. Similar to the benchmark created from the NQ dataset, we randomly sample 50K unique passages to constitute the initial D0 corpus, and five more corpora, each with 10K passages (see Section 2.2 for more details). **In Table 3 (page 13), we report our results on the MS Marco benchmark for DSI++**. In summary, we show that ER with generative memory improves the overall performance of the retrieval task, reducing forgetting of previously indexed documents and enabling forward transfer to newly indexed documents. **We show that our results hold across two datasets – Natural Questions and MS MARCO, thus, showcasing the generalizability of our approach**.

Let us know if we answer your questions convincingly. We are happy to answer any follow-up questions!


**References**

[Mehta et al. (2021)] Mehta, Sanket Vaibhav, Darshan Patil, Sarath Chandar, and Emma Strubell. "An empirical investigation of the role of pre-training in lifelong learning." arXiv preprint arXiv:2112.09153 (2021).

[Sun et al. (2020)] Sun, Fan-Keng, Cheng-Hao Ho, and Hung-Yi Lee. "LAMOL: LAnguage MOdeling for Lifelong Language Learning." In International Conference on Learning Representations. 2020.

[De Lange et al. (2021)] De Lange, Matthias, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. "A continual learning survey: Defying forgetting in classification tasks." IEEE transactions on pattern analysis and machine intelligence 44, no. 7 (2021): 3366-3385.

[Nguyen et al. (2016)] Nguyen, Tri, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. "MS MARCO: A human generated machine reading comprehension dataset." In CoCo@ NIPs. 2016.

From: ICLR.cc/2023/Conference/Paper5140/Reviewer_mfHG
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Authors show differentiable search indicies when continual indexing of new documents leads to considerable forgetting of previously indexed documents. The authors propose two approaches to mitigate the forgetting in a continual learning scenario. A) first focuses on modifying the training dynamics. Flatter minima implicitly alleviates forgetting, so they advocate doing that. In their experiments they got +12% (I.e. the model stably memorizes more documents) by doing this. B) Create a novel continual indexing benchmark based on Natural Questions. The term their approach DSI++ 
strength_and_weaknesses: Strengths: 

1. The authors work on an important idea. The original DSI is an exciting idea of having informatin retrieval encapsulated into a complete end-to-end learnable paradigm of differentiable and back-propagation.  

2. The paper is fairly clear in their presentation and the authors clearly state the parameters of their experiments – metrics, data, training and evaluation protocols. Based on the experiments (table 1) Authors discuss various approaches to glean more into their approach. 

3. Releasing a dataset benchmark in this area might prove to be valuable. 

Weaknesses: 

1. My main point of concerns with this paper is the lack of comparisons with relavent prior art. E.g. papers [1-3] discuss ways to alter predictions at test-time (without re-training the retriever) using external retrievable memory. The authors fail to compare with these papers making it hard for a reviewer (like me) or a reader in this fast-evolving research field to understand the magnitude of contribution of this work in relation to other papers in this area. I might concead to the point that some of these works are not in continual learning domain, however I feel that should not be the main reason to not show comparisons. 

2. Another set of experiments that might have made the paper stronger is to implement these changes in more architectures besides just T5 and demonstrate to the reader that the approach is generalizable and it works across various approaches.  

3. Lots of missing citations.  

[1] Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering 
[2] Generalization through Memorization: Nearest Neighbor Language Models 
[3] more cited by and citing these papers. 

 
clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear in their presentation and the authors clearly state the parameters of their experiments – metrics, data, training and evaluation protocols. Based on the experiments (table 1) Authors discuss various approaches to glean more into their approach. 

no code is provided to my knowledge and i am unsure if the authors plan to release the code.
summary_of_the_review: The authors tackle an important problem and show interesting results. However, I vote reject for ICLR 2023 based on the weaknesses in Experimental section. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5140/Reviewer_YCgn
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors proposed continually learning the Differentiable search index, such that it continuously updates models for answering queries related to new and old documents. For this, they have targeted two aspects of forgetting 1) Implicit forgetting and 2) Explicit forgetting. The methods are based on the existing work Sharpness aware minimization (SAM) for implicit forgetting and generating the queries for old and new documents to use during continual indexing. 

strength_and_weaknesses: Strengths: 
1) DSI++ verifies the SAM-based stable memorization on T5 base, large, and XL by optimizing selectively for the flatter loss basins. This way, the model memorizes $12%$ more documents. 

2) The authors have also proposed the generative experience rehearsal. This way, the average hit is increased by $21%$ over baselines. The proposed generative experience rehearsal does not retain the original queries and performs better than the episodic memory-based replay.

3) The paper is well written and experimental analysis is shown to provide evidence for continually updating DSI++.

Weakness and Questions:
1) The paper's novelty is unclear. It has used the existing approaches - Sharpness aware minimization (SAM) and generative methods for episodic replay (Sun et al.) on T5 model.

2) Why do we not have access to the incoming stream of new documents? The experimental setting needs to be clarified.

3) It will be great to benchmark additional continual learning-based methods like Parameter isolation* and model expansion methods like LWF for DSI. How do they compare with SAM and generative ER?

*De Lange, Matthias, et al. "A continual learning survey: Defying forgetting in classification tasks." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3366-3385.

4) Additional datasets benchmarks can be used/constructed to strengthen the argument.

clarity,_quality,_novelty_and_reproducibility: The paper is written well and clear. Novelty is not very strong. They have used the NQ dataset and made splits for the continual learning process. The data spilts will be shared in future. No code is provided but parameters are listed to replicate the experiments. 
summary_of_the_review: DSI has been celebrated for an efficient memorization based retrieval and I greatly welcome the efforts in the direction of continually updating DSI. I feel more experiments with continual learning methods catering to language models will make a solid contribution.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5140/Reviewer_uo3e
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: In the context of information retrieval, this paper proposes techniques to overcome the forgetting issue with Differentiable Search Index (that encodes document information within the model parameters) by using standard techniques borrowed from continuous learning (sampling past examples / SAM optimization) and IR data augmentation (generating queries from documents). Experiments conducted on 

strength_and_weaknesses: This paper is the first to study the problem of updating DSI in IR and the results confirm that standard techniques do work in practice. It is nice to have actual experiments showing how severe is the problem (and how to solve it).

The experiments are (mostly) well conducted and report expected results.

The only experiment part which is not really needed is the comparison between naively structured IDs (generating the document ID) and unstructured atomic (classification task), where the first is obviously harder (as shown in the original paper). It would be better to look at methods trying to bring some structure to the document identifiers (including the one in the original paper, i.e. Semantic String Docid) -- which would definitely help to prevent forgetting. This is not a major concern however since these works are orthogonal, but it would have been good to see if the degradation is as severe in those cases.

More importantly, the paper also fully ignores the recent data augmentation works that generate a query from a document to generate training samples (Promptagator: Few-shot Dense Retrieval From 8 Examples, arXiv 2022 ; Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation, arXiv 2022; InPars: Unsupervised Dataset Generation for Information Retrieval, SIGIR 2022).

Another downside is the relatively low number of documents in Natural Questions (50K), which is way below other datasets (e.g. MS Marco with 8M documents), and clearly limits the overall statements made in this paper.

The paper did not report the performance of DSI when the full dataset is used for training (which would be a strong baseline).

clarity,_quality,_novelty_and_reproducibility: The novelty of the paper is rather low (in terms of techniques) - and lies in being the first to look at the problem of forgetting for DSI.

The writing quality is good, and the paper is well organized apart from a few unclear parts:
- a good part of experimental results is described before the experimental protocol, which does not help to contextualize the presented results (e.g. in section 2.4);
- in section 3, what "explicitly adding documents" mean?

summary_of_the_review: This paper discusses the problem of the forgetting issue in DSI -- given the novelty of DSIs, this work is the first to look at this problem. The solutions are taken from the literature and evaluated (on a small dataset compared to standard IR settings) showing that they help mitigate the problem (but with a naive DSI where document identifiers are arbitrary). The results are however quite limited in scope (given the size of the dataset, and the fact that is a unique dataset -- as well as the fact that the DSI studied is only one of the options that have been presented so far).
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5140/Reviewer_9oZM
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
strength_and_weaknesses: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
clarity,_quality,_novelty_and_reproducibility: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
summary_of_the_review: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold