Original: A Neural Corpus Indexer for Document Retrieval
Revised: A Neural Corpus Indexer for Document Retrieval

Original: Abstract
Revised: Abstract

Original: Current state-of-the-art document retrieval solutions mainly follow an indexretrieve paradigm, where the index is hard to be directly optimized for the final retrieval target.
Revised: Current state-of-the-art document retrieval solutions mainly follow an indexretrieve paradigm, where the index is hard to be directly optimized for the final retrieval target.

Original: In this paper, we aim to show that an end-to-end deep neural network unifying training and indexing stages can significantly improve the recall performance of traditional methods.
Revised: In this paper, we aim to show that an end-to-end deep neural network unifying training and indexing stages can significantly improve the recall performance of traditional methods.

Original: To this end, we propose Neural Corpus Indexer (NCI), a sequence-to-sequence network that generates relevant document identifiers directly for a designated query.
Revised: To this end, we propose Neural Corpus Indexer (NCI), a sequence-to-sequence network that generates relevant document identifiers directly for a designated query.

Original: To optimize the recall performance of
Revised: To optimize the recall performance of

Original: NCI, we invent a prefix-aware weight-adaptive decoder architecture, and leverage tailored techniques including query generation, semantic document identifiers, and consistency-based regularization.
Revised: NCI, we invent a prefix-aware weight-adaptive decoder architecture, and leverage tailored techniques including query generation, semantic document identifiers, and consistency-based regularization.

Original: Empirical studies demonstrated the superiority of NCI on two commonly used academic benchmarks, achieving +17.6% and +16.8% relative enhancement for Recall@1 on NQ320 k dataset and R-Precision on TriviaQA dataset, respectively, compared to the best baseline method.
Revised: Empirical studies demonstrated the superiority of NCI on two commonly used academic benchmarks, achieving +21.4% and +16.8% relative enhancement for Recall@1 on NQ320 k dataset and R-Precision on TriviaQA dataset, respectively, compared to the best baseline method.

Original:  Introduction
Revised:  Introduction

Original: Document retrieval and ranking are two key stages for a standard web search engine [56, 34].
Revised: Document retrieval and ranking are two key stages for a standard web search engine [56, 34].

Original: First, the document retrieval stage retrieves candidate documents relevant to the query, and then, the ranking stage gives a more precise ranking score for each document.
Revised: First, the document retrieval stage retrieves candidate documents relevant to the query, and then, the ranking stage gives a more precise ranking score for each document.

Original: The ranking stage is often fulfilled by a deep neural network, taking each pair of query and document as input and predicting their relevance score.
Revised: The ranking stage is often fulfilled by a deep neural network, taking each pair of query and document as input and predicting their relevance score.

Original: Nevertheless, a precise ranking model is very costly, while typically only a hundred or thousand candidates per query are affordable in an online system.
Revised: Nevertheless, a precise ranking model is very costly, while typically only a hundred or thousand candidates per query are affordable in an online system.

Original: As a result, the recall performance of the document retrieval stage is very crucial to the effectiveness of web search engines.
Revised: As a result, the recall performance of the document retrieval stage is very crucial to the effectiveness of web search engines.

Original: Existing document retrieval methods can be divided into two categories, namely term-based and semantic-based approaches [22].
Revised: Existing document retrieval methods can be divided into two categories, namely term-based and semantic-based approaches [22].

Original: Term-based retrieval approaches [9, 59] build an inverted index for the entire web corpus, but they hardly capture document semantics and fail to retrieve similar documents in different wordings.
Revised: Term-based retrieval approaches [9, 59] build an inverted index for the entire web corpus, but they hardly capture document semantics and fail to retrieve similar documents in different wordings.

Original: Thus, semantic-based approaches [56, 36] are proposed to alleviate this discrepancy.
Revised: Thus, semantic-based approaches [56, 36] are proposed to alleviate this discrepancy.

Original: First, they learn dense representations for both queries and documents through a twin-tower architecture; then Approximate Nearest Neighbor (ANN) search is applied to retrieve relevant documents for the designated query.
Revised: First, they learn dense representations for both queries and documents through a twin-tower architecture; then Approximate Nearest Neighbor (ANN) search is applied to retrieve relevant documents for the designated query.

Original: Despite of their success in real applications, these approaches can not fully leverage the power of deep neural networks for the following reasons.
Revised: Despite of their success in real applications, these approaches can not fully leverage the power of deep neural networks for the following reasons.

Original: First, a single embedding vector has limited capacity to memorize all semantics in a document, and it performs even worse than term-based methods in the applications that heavily rely on exact match [37].
Revised: First, a single embedding vector has limited capacity to memorize all semantics in a document, and it performs even worse than term-based methods in the applications that heavily rely on exact match [37].

Original: Second, the model is unable to incorporate deep query-document interactions.
Revised: Second, the model is unable to incorporate deep query-document interactions.

Original: Because
Revised: Because

Original: ANN algorithms theoretically require a strong assumption for the Euclidean space, we have to adopt simple functions such as cosine similarity to capture the query-document interactions [20].
Revised: ANN algorithms theoretically require a strong assumption for the Euclidean space, we have to adopt simple functions such as cosine similarity to capture the query-document interactions [20].

Original: Given the above limitations, several research works have explored end-to-end models that directly retrieve relevant candidates without using an explicit index.
Revised: Given the above limitations, several research works have explored end-to-end models that directly retrieve relevant candidates without using an explicit index.

Original: Gao et al.
Revised: Gao et al.

Original: [20] proposed a Deep Retrieval (DR) framework for item recommendation, which learned a retrievable structure with historical user-item interactions.
Revised: [20] proposed a Deep Retrieval (DR) framework for item recommendation, which learned a retrievable structure with historical user-item interactions.

Original: Nevertheless, it is more challenging to design a universal model for semantic text retrieval, as we need to leverage the power of both pre-trained language models and deep retrieval networks simultaneously.
Revised: Nevertheless, it is more challenging to design a universal model for semantic text retrieval, as we need to leverage the power of both pre-trained language models and deep retrieval networks simultaneously.

Original: Tay et al.
Revised: Tay et al.

Original: [50] proposed Differentiable Search Index (DSI), a text-to-text model that maps queries directly to relevant docids.
Revised: [50] proposed Differentiable Search Index (DSI), a text-to-text model that maps queries directly to relevant docids.

Original: To the best of our knowledge, this is the first attempt to propose a differentiable index for semantic search.
Revised: To the best of our knowledge, this is the first attempt to propose a differentiable index for semantic search.

Original: However, the vanilla transformer decoder in DSI does not fully leverage the hierarchical structures of document identifiers, and the model is pruned to over-fitting with limited training data.
Revised: However, the vanilla transformer decoder in DSI does not fully leverage the hierarchical structures of document identifiers, and the model is pruned to over-fitting with limited training data.

Original: Furthermore, Bevilacqua et al.
Revised: Furthermore, Bevilacqua et al.

Original: [4] proposed
Revised: [4] proposed

Original: SEAL by leveraging all n-grams in a passage as its identifiers.
Revised: SEAL by leveraging all n-grams in a passage as its identifiers.

Original: But for long documents, it is hard to enumerate all possible n-grams.
Revised: But for long documents, it is hard to enumerate all possible n-grams.

Original: In general, the recall performance of end-to-end document retrieval remains a large room to be improved.
Revised: In general, the recall performance of end-to-end document retrieval remains a large room to be improved.

Original: In this paper, we show that the traditional text retrieval frameworks can be fundamentally changed by a unified deep neural network with tailored designs.
Revised: In this paper, we show that the traditional text retrieval frameworks can be fundamentally changed by a unified deep neural network with tailored designs.

Original: To this end, we propose a Neural Corpus Indexer (NCI), which supports end-to-end document retrieval by a sequence-to-sequence neural network.
Revised: To this end, we propose a Neural Corpus Indexer (NCI), which supports end-to-end document retrieval by a sequence-to-sequence neural network.

Original: The model takes a user query as input, generates the query embedding through the encoder, and outputs the identifiers of relevant documents using the decoder.
Revised: The model takes a user query as input, generates the query embedding through the encoder, and outputs the identifiers of relevant documents using the decoder.

Original: It can be trained by both ground-truth and augmented query-document pairs.
Revised: It can be trained by both ground-truth and augmented query-document pairs.

Original: During inference, the top N documents are retrieved via beam search based on the decoder.
Revised: During inference, the top N documents are retrieved via beam search based on the decoder.

Original: Designing and training such a model is non-trivial, so we propose several crucial techniques to ensure its effectiveness.
Revised: Designing and training such a model is non-trivial, so we propose several crucial techniques to ensure its effectiveness.

Original: First, to get sufficient query-document pairs for training, we leverage a query generation network to obtain possible pairs of queries and documents.
Revised: First, to get sufficient query-document pairs for training, we leverage a query generation network to obtain possible pairs of queries and documents.

Original: Second, we utilize the hierarchical k -means algorithm to generate a semantic identifier for each document.
Revised: Second, we utilize the hierarchical k -means algorithm to generate a semantic identifier for each document.

Original: Third, we design a prefix-aware weight-adaptive decoder to replace the vanilla one in a sequence-to-sequence architecture.
Revised: Third, we design a prefix-aware weight-adaptive decoder to replace the vanilla one in a sequence-to-sequence architecture.

Original: Specifically, the same token will be assigned different embedding vectors at different positions in the identifiers, while another transformer-based adaptive module is applied to the classification weights for token prediction in the context of a certain prefix.
Revised: Specifically, the same token will be assigned different embedding vectors at different positions in the identifiers, while another transformer-based adaptive module is applied to the classification weights for token prediction in the context of a certain prefix.

Original: This makes the classifiers customized to different prefixes when decoding along the hierarchical tree structure.
Revised: This makes the classifiers customized to different prefixes when decoding along the hierarchical tree structure.

Original: Besides, a consistency-based regularization loss is taken for training both encoder and decoder networks to mitigate the over-fitting problem.
Revised: Besides, a consistency-based regularization loss is taken for training both encoder and decoder networks to mitigate the over-fitting problem.

Original: Our NCI design solves the limitations of traditional index-retrieve pipelines from multiple perspectives.
Revised: Our NCI design solves the limitations of traditional index-retrieve pipelines from multiple perspectives.

Original: On one hand, a whole neural network model replaces the traditional inverted index or vector search solutions.
Revised: On one hand, a whole neural network model replaces the traditional inverted index or vector search solutions.

Original: It can be optimized end-to-end using realistic query-document pairs, which fully captures both term-based and semantic-based features and is adaptive to the changing of workloads.
Revised: It can be optimized end-to-end using realistic query-document pairs, which fully captures both term-based and semantic-based features and is adaptive to the changing of workloads.

Original: On the other hand, the model is able to capture deep interactions between queries and documents via the encoder-decoder attention, which enlarges the capacity of vector-based representations.
Revised: On the other hand, the model is able to capture deep interactions between queries and documents via the encoder-decoder attention, which enlarges the capacity of vector-based representations.

Original: Moreover,
Revised: Moreover,

Original: NCI achieves much better ranking results than ANN-based approaches as it is optimized directly by the final target.
Revised: NCI achieves much better ranking results than ANN-based approaches as it is optimized directly by the final target.

Original: Thus, it can be served as an end-to-end retrieval solution while releasing the burden of re-ranking for a long candidate list.
Revised: Thus, it can be served as an end-to-end retrieval solution while releasing the burden of re-ranking for a long candidate list.

Original: In addition to the superior performance, the invention of Neural Corpus Indexer is also promising from the perspective of system design.
Revised: In addition to the superior performance, the invention of Neural Corpus Indexer is also promising from the perspective of system design.

Original: As nowadays, ranking and query-answering modules are already implemented by neural networks, NCI finishes the last piece of puzzle for the next-generation information retrieval system based on a unified differentiable model architecture.
Revised: As nowadays, ranking and query-answering modules are already implemented by neural networks, NCI finishes the last piece of puzzle for the next-generation information retrieval system based on a unified differentiable model architecture.

Original: This reduces the dependency among different sub-modules, while the processes of system deployment and maintenance could be greatly eased.
Revised: This reduces the dependency among different sub-modules, while the processes of system deployment and maintenance could be greatly eased.

Original: Our contributions are highlighted as follows.
Revised: Our contributions are highlighted as follows.

Original: • For the first time, we demonstrate that an end-to-end differentiable document retrieval model can significantly outperform both inverted index and dense retrieval solutions.
Revised: • For the first time, we demonstrate that an end-to-end differentiable document retrieval model can significantly outperform both inverted index and dense retrieval solutions.

Original: This finding will inspire research on further steps towards the next-generation search systems, for instance, unifying informational retrieval, ranking, and question answering in a single differentiable framework.
Revised: This finding will inspire research on further steps towards the next-generation search systems, for instance, unifying informational retrieval, ranking, and question answering in a single differentiable framework.

Original: • We design a sequence-to-sequence model, named Neural Corpus Indexer (NCI), which generates relevant document identifiers directly for a specific query.
Revised: • We design a sequence-to-sequence model, named Neural Corpus Indexer (NCI), which generates relevant document identifiers directly for a specific query.

Original: In our experiments, the proposed NCI model improves the state-of-the-art performance of existing methods by a significant margin, achieving +17.6%
Revised: In our experiments, the proposed NCI model improves the state-of-the-art performance of existing methods by a significant margin, achieving +21.4%

Original: and +16.8% relative enhancement for Recall@1 on NQ320 k dataset and
Revised: and +16.8% relative enhancement for Recall@1 on NQ320 k dataset and

Original: R-Precision on TriviaQA dataset, respectively.
Revised: R-Precision on TriviaQA dataset, respectively.

Original: Also, NCI itself achieves a competitive MRR score without using an explicit ranking model.
Revised: Also, NCI itself achieves a competitive MRR score without using an explicit ranking model.

Original: • We propose a novel decoder architecture, namely prefix-aware weight-adaptive (PAWA) decoder, to generate document identifiers.
Revised: • We propose a novel decoder architecture, namely prefix-aware weight-adaptive (PAWA) decoder, to generate document identifiers.

Original: As verified by ablation studies, this invention is very crucial for NCI to achieve an outstanding performance.
Revised: As verified by ablation studies, this invention is very crucial for NCI to achieve an outstanding performance.

Original: Moreover, query generation, semantic document identifiers, and consistency-based regularization are all accountable for the superior capability of Neural Corpus Indexer.
Revised: Moreover, query generation, semantic document identifiers, and consistency-based regularization are all accountable for the superior capability of Neural Corpus Indexer.

Original:  Related work
Revised:  Related work

Original: In this section, we briefly introduce the related works and leave more discussionsabout the traditional web search techniques inthe Appendix A.
Revised: In this section, we briefly introduce the related works and leave more discussions in Appendix A.

Original: Sparse retrieval.
Revised: Sparse retrieval.

Original: Traditional document retrieval methods are based on Sparse Retrieval , which is built upon inverted index with term matching metrics such as TF-IDF [45], query likelihood [33] or BM25 [44].
Revised: Traditional document retrieval methods are based on Sparse Retrieval , which is built upon inverted index with term matching metrics such as TF-IDF [45], query likelihood [33] or BM25 [44].

Original: In industry-scale web search, BM25 is a difficult-to-beat baseline owing to its outstanding trade-off between accuracy and efficiency.
Revised: In industry-scale web search, BM25 is a difficult-to-beat baseline owing to its outstanding trade-off between accuracy and efficiency.

Original: In recent years, there are some attempts to incorporate the power of neural networks into inverted index.
Revised: In recent years, there are some attempts to incorporate the power of neural networks into inverted index.

Original: The Standalone Neural Ranking
Revised: The Standalone Neural Ranking

Original: Model (SNRM) [57] learns high-dimensional sparse representations for query and documents, which enables the construction of inverted index for efficient document retrieval.
Revised: Model (SNRM) [57] learns high-dimensional sparse representations for query and documents, which enables the construction of inverted index for efficient document retrieval.

Original: Doc2Query [41] predicts relevant queries to augment the content of each document before building the BM25 index, and
Revised: Doc2Query [41] predicts relevant queries to augment the content of each document before building the BM25 index, and

Original: DocT5Query [40] improves the performance of query generation by the pre-trained language model
Revised: DocT5Query [40] improves the performance of query generation by the pre-trained language model

Original: T5 [5].
Revised: T5 [5].

Original: Furthermore, DeepCT [9] calculates context-aware term importance through neural networks to improve the term matching metrics of BM25.
Revised: Furthermore, DeepCT [9] calculates context-aware term importance through neural networks to improve the term matching metrics of BM25.

Original: Dense retrieval.
Revised: Dense retrieval.

Original: Another line of research lies in Dense Retrieval , which presents query and documents in dense vectors and models their similarities with inner product or cosine similarity.
Revised: Another line of research lies in Dense Retrieval , which presents query and documents in dense vectors and models their similarities with inner product or cosine similarity.

Original: These methods benefit from recent progresses of pre-trained language models, such as BERT [14] and RoBERTa [35] to obtain dense representations for queries and documents.
Revised: These methods benefit from recent progresses of pre-trained language models, such as BERT [14] and RoBERTa [35] to obtain dense representations for queries and documents.

Original: At inference time, efficient Approximate
Revised: At inference time, efficient Approximate

Original: Nearest Neighbor (ANN) search algorithms, such as k-dimensional trees [3], locality-sensitive hashing [10], and graph-based indexes (e.g., HNSW [38], DiskANN [27] and SPANN [7]) can be utilized to retrieve relevant documents within a sublinear time.
Revised: Nearest Neighbor (ANN) search algorithms, such as k-dimensional trees [3], locality-sensitive hashing [10], and graph-based indexes (e.g., HNSW [38], DiskANN [27] and SPANN [7]) can be utilized to retrieve relevant documents within a sublinear time.

Original: Besides, Luan et al.
Revised: Besides, Luan et al.

Original: [37] analyze the limited capacity of dual encoders, and propose a combination of sparse and dense retrieval methods with multi-vector encoding to achieve better search quality.
Revised: [37] analyze the limited capacity of dual encoders, and propose a combination of sparse and dense retrieval methods with multi-vector encoding to achieve better search quality.

Original: Autoregressive retrieval.
Revised: Autoregressive retrieval.

Original: The other way to approach retrieval is utilizing an end-to-end autoregressive model.
Revised: The other way to approach retrieval is utilizing an end-to-end autoregressive model.

Original: Firstly, several efforts have been done on entity linking [13, 12, 11], which can be regarded as a special type of retrieval task, e.g. , using an entity to ask the posed question.
Revised: Firstly, several efforts have been done on entity linking [13, 12, 11], which can be regarded as a special type of retrieval task, e.g. , using an entity to ask the posed question.

Original: Recently, different from the entity linking task, Tay et al.
Revised: Recently, different from the entity linking task, Tay et al.

Original: [50] proposed the DSI (differentiable search index) model to generate relevant document identifiers directly corresponding to the query.
Revised: [50] proposed the DSI (differentiable search index) model to generate relevant document identifiers directly corresponding to the query.

Original: Bevilacqua et al.
Revised: Bevilacqua et al.

Original: [4] employed the autoregressive model to generate relevant words for a query and utilize the generated string to retrieve relevant documents.
Revised: [4] employed the autoregressive model to generate relevant words for a query and utilize the generated string to retrieve relevant documents.

Original: Besides, the Deep Retrieval (DR) [20] approach for recommendation is also related to this category, which learns a deep retrievable network with user-item clicks and gets rid of the ANN algorithms based on the Euclidean space assumption.
Revised: Besides, the Deep Retrieval (DR) [20] approach for recommendation is also related to this category, which learns a deep retrievable network with user-item clicks and gets rid of the ANN algorithms based on the Euclidean space assumption.

Original: Pre-trained language models.
Revised: Pre-trained language models.

Original: Recently, pre-trained Language Models (LMs), such as BERT [14] and RoBERTa [35], have led to a revolution in web search techniques.
Revised: Recently, pre-trained Language Models (LMs), such as BERT [14] and RoBERTa [35], have led to a revolution in web search techniques.

Original: The representation vectors for all documents can be calculated and indexed offline.
Revised: The representation vectors for all documents can be calculated and indexed offline.

Original: In the online serving stage, it calculates the representation vector for the input query, and applies a crossing layer to calculate the relevance score between each query and document pair.
Revised: In the online serving stage, it calculates the representation vector for the input query, and applies a crossing layer to calculate the relevance score between each query and document pair.

Original: The crossing layer usually adopts simple operators such as cosine similarity or a single feed-forward layer to retain a high efficiency.
Revised: The crossing layer usually adopts simple operators such as cosine similarity or a single feed-forward layer to retain a high efficiency.

Original: Gao et al.
Revised: Gao et al.

Original: [16] found that a standard LMs’ internal attention structure is not ready-to-use for dense encoders and proposed the Condenser to improve the performance of dense retrieval.
Revised: [16] found that a standard LMs’ internal attention structure is not ready-to-use for dense encoders and proposed the Condenser to improve the performance of dense retrieval.

Original: Moreover, ANCE [54] leverages hard negatives to improve the effectiveness of contrastive learning, which generates better text representations for the retrieval tasks.
Revised: Moreover, ANCE [54] leverages hard negatives to improve the effectiveness of contrastive learning, which generates better text representations for the retrieval tasks.

Original:  Neural corpus indexer
Revised:  Neural corpus indexer

Original: The neural corpus indexer (NCI) is a sequence-to-sequence neural network model.
Revised: The neural corpus indexer (NCI) is a sequence-to-sequence neural network model.

Original: The model takes a query as input and outputs the most relevant document identifier (docid) , which can be trained by a large collection of <query, docid> pairs.
Revised: The model takes a query as input and outputs the most relevant document identifier (docid) , which can be trained by a large collection of <query, docid> pairs.

Original: The documents are encoded into semantic docids by the hierarchical k -means algorithm [23], which makes similar documents have “close” identifiers in the hierarchical tree.
Revised: The documents are encoded into semantic docids by the hierarchical k -means algorithm [23], which makes similar documents have “close” identifiers in the hierarchical tree.

Original: As shown in Figure 1, NCI is composed of three components, including Query Generation , Encoder , and Prefix-Aware Weight-Adaptive (PAWA) Decoder .
Revised: As shown in Figure 1, NCI is composed of three components, including Query Generation , Encoder , and Prefix-Aware Weight-Adaptive (PAWA) Decoder .

Original: Query generation is implemented by a sequence-to-sequence transformer model [52] that takes as input the document terms and produces a query as output [41].
Revised: Query generation is implemented by a sequence-to-sequence transformer model [52] that takes as input the document terms and produces a query as output [41].

Original: The encoder, following the standard transformer architecture, is composed of M 1 stacked transformer blocks, which outputs the representation for an input query.
Revised: The encoder, following the standard transformer architecture, is composed of M 1 stacked transformer blocks, which outputs the representation for an input query.

Original: For the decoder network, we stack M 2 transformer layers.
Revised: For the decoder network, we stack M 2 transformer layers.

Original: To better align with the hierarchical nature of the semantic identifiers, we propose a weight adaptation mechanism based on another transformer to make the decoder aware of semantic prefixes.
Revised: To better align with the hierarchical nature of the semantic identifiers, we propose a weight adaptation mechanism based on another transformer to make the decoder aware of semantic prefixes.

Original: At inference time, the top
Revised: At inference time, the top

Original: N relevant documents can be easily obtained via beam search.
Revised: N relevant documents can be easily obtained via beam search.

Original: Due to the hierarchical property of semantic identifiers, it is easy to constrain the beam search on the prefix tree so that only valid identifiers will be generated.
Revised: Due to the hierarchical property of semantic identifiers, it is easy to constrain the beam search on the prefix tree so that only valid identifiers will be generated.

Original: 3.1 Representing document with semantic identifiers
Revised: 3.1 Representing document with semantic identifiers

Original: NCI generates document identifiers solely based on the input query without explicit document content, which is difficult when the size of the corpus is very large.
Revised: NCI generates document identifiers solely based on the input query without explicit document content, which is difficult when the size of the corpus is very large.

Original: Thus, we aim to inject useful priors into the identifiers so that the semantic information of documents can be incorporated in the decoding process.
Revised: Thus, we aim to inject useful priors into the identifiers so that the semantic information of documents can be incorporated in the decoding process.

Original: In other words, we hope the documents with similar semantics have close docids to facilitate the learning process of NCI.
Revised: In other words, we hope the documents with similar semantics have close docids to facilitate the learning process of NCI.

Original: To achieve this, we leverage the hierarchical k -means algorithm to encode documents.
Revised: To achieve this, we leverage the hierarchical k -means algorithm to encode documents.

Original: As shown in Figure 1(a), given a collection of documents to be indexed, all documents are first classified into k clusters by using their representations encoded by BERT [14].
Revised: As shown in Figure 1(a), given a collection of documents to be indexed, all documents are first classified into k clusters by using their representations encoded by BERT [14].

Original: For cluster with more than c documents, the k -means algorithm is applied recursively.
Revised: For cluster with more than c documents, the k -means algorithm is applied recursively.

Original: For each cluster containing c documents or less, each document is assigned a number starting from 0 to at most c -1.
Revised: For each cluster containing c documents or less, each document is assigned a number starting from 0 to at most c -1.

Original: In this way, we organize all documents into a tree structure T with root r 0 .
Revised: In this way, we organize all documents into a tree structure T with root r 0 .

Original: Each document is associated with one leaf node with a deterministic routing path l = { r 0 , r 1 , ..., r m } from the root, where r i ∈ [0 , k ) represents the internal cluster index for level i , and r m ∈ [0 , c ) is the leaf node.
Revised: Each document is associated with one leaf node with a deterministic routing path l = { r 0 , r 1 , ..., r m } from the root, where r i ∈ [0 , k ) represents the internal cluster index for level i , and r m ∈ [0 , c ) is the leaf node.

Original: The semantic identifier for a document is concatenated by the node indices along the path from root to its corresponding leaf node.
Revised: The semantic identifier for a document is concatenated by the node indices along the path from root to its corresponding leaf node.

Original: For documents with similar semantics, the prefixes of their corresponding identifiers are likely to be the same.
Revised: For documents with similar semantics, the prefixes of their corresponding identifiers are likely to be the same.

Original: For simplicity, we set k = 30 and c = 30 in all experiments, leaving the optimization of these hyper-parameters to future work.
Revised: For simplicity, we set k = 30 and c = 30 in all experiments, leaving the optimization of these hyper-parameters to future work.

Original: The detailed procedure of hierarchical k -means will be described in Algorithm 1 in the Appendix B.2.
Revised: The detailed procedure of hierarchical k -means will be described in Algorithm 1 in the Appendix B.2.

Original: 3.2 Query generation
Revised: 3.2 Query generation

Original: One challenge of generating document identifiers by single query input is how to make the identifiers aware of the document semantics.
Revised: One challenge of generating document identifiers by single query input is how to make the identifiers aware of the document semantics.

Original: Since the content of each document is not explicitly known at inference, it must be incorporated into the model parameters during training.
Revised: Since the content of each document is not explicitly known at inference, it must be incorporated into the model parameters during training.

Original: To facilitate the training process, we generate a bunch of queries with a query generation module and bind the information of document content through training the sequence-to-sequence model with generated queries and their corresponding document identifiers.
Revised: To facilitate the training process, we generate a bunch of queries with a query generation module and bind the information of document content through training the sequence-to-sequence model with generated queries and their corresponding document identifiers.

Original: In NCI, we utilize two kinds of augmented queries:
Revised: In NCI, we utilize two kinds of augmented queries:

Original: DocT5Query.
Revised: DocT5Query.

Original: We adopt a standard sequence-to-sequence transformer [52] based on the implementation of DocT5Query [1] pre-trained by a large query-document corpus.
Revised: We adopt a standard sequence-to-sequence transformer [52] based on the implementation of DocT5Query [1] pre-trained by a large query-document corpus.

Original: It takes as input the document terms and produces relevant queries via random sampling.
Revised: It takes as input the document terms and produces relevant queries via random sampling.

Original: Note that we use random sampling instead of beam search to ensure the diversity of generated queries.
Revised: Note that we use random sampling instead of beam search to ensure the diversity of generated queries.

Original: Document As Query.
Revised: Document As Query.

Original: Like DSI [50], we also utilize the first 64 terms for each document as queries.
Revised: Like DSI [50], we also utilize the first 64 terms for each document as queries.

Original: Besides, we randomly selected 10 groups of 64 consecutive terms from the whole article as additional queries.
Revised: Besides, we randomly selected 10 groups of 64 consecutive terms from the whole article as additional queries.

Original: This makes the NCI model aware of the semantic meaning of each document.
Revised: This makes the NCI model aware of the semantic meaning of each document.

Original: 3.3 Prefix-aware weight-adaptive decoder
Revised: 3.3 Prefix-aware weight-adaptive decoder

Original: Given an input query x , the probability of generating a document identifier can be written as:
Revised: Given an input query x , the probability of generating a document identifier can be written as:

Original: [Equation]
Revised: [Equation]

Original: where r i is the i -th token in the current identifier; x is the representation output from encoder; θ denotes the total parameters and θ i is the parameter for the i -th step.
Revised: where r i is the i -th token in the current identifier; x is the representation output from encoder; θ denotes the total parameters and θ i is the parameter for the i -th step.

Original: This probability can be modeled by a transformer-based decoder.
Revised: This probability can be modeled by a transformer-based decoder.

Original: For an internal node with level i , the probability is calculated by:
Revised: For an internal node with level i , the probability is calculated by:

Original: [Equation]
Revised: [Equation]

Original: Here h i is the hidden representation for step i , which is calculated by a multi-head attention over encoder representation x and token representations of previous decoding steps.
Revised: Here h i is the hidden representation for step i , which is calculated by a multi-head attention over encoder representation x and token representations of previous decoding steps.

Original: The linear classification weight is denoted by W ∈ R d × v , d is the hidden dimension size and v is the vocabulary size of identifiers.
Revised: The linear classification weight is denoted by W ∈ R d × v , d is the hidden dimension size and v is the vocabulary size of identifiers.

Original: As the encoder and decoder utilize distinct vocabulary spaces, we do not share the embedding space for their tokens.
Revised: As the encoder and decoder utilize distinct vocabulary spaces, we do not share the embedding space for their tokens.

Original: Different from a standard decoding task, the meanings of the same token appearing at different places of the same identifier are different, as they correspond to different clusters in the hierarchical tree structure.
Revised: Different from a standard decoding task, the meanings of the same token appearing at different places of the same identifier are different, as they correspond to different clusters in the hierarchical tree structure.

Original: For instance, the “ 5 2 ” and “ 5 3 ” of the same identifier “ 3 1 5 2 5 3 ” correspond to different semantic meanings.
Revised: For instance, the “ 5 2 ” and “ 5 3 ” of the same identifier “ 3 1 5 2 5 3 ” correspond to different semantic meanings.

Original: Moreover, the same token in the same position may have different semantics with different prefixes.
Revised: Moreover, the same token in the same position may have different semantics with different prefixes.

Original: For example, in identifiers “ 1 1 1 2 5 3 ” and ” 2 1 4 2 5 3 ”, the same token “ 5 3 ” has different semantics in two different identifiers, as they are routed from different prefix paths.
Revised: For example, in identifiers “ 1 1 1 2 5 3 ” and ” 2 1 4 2 5 3 ”, the same token “ 5 3 ” has different semantics in two different identifiers, as they are routed from different prefix paths.

Original: These two properties of the hierarchical semantic identifiers motivate us to design the novel
Revised: These two properties of the hierarchical semantic identifiers motivate us to design the novel

Original: P refix- A ware W eight- A daptor (PAWA) decoder.
Revised: P refix- A ware W eight- A daptor (PAWA) decoder.

Original: Unlike a standard transformer decoder, the probabilities at different tree levels, such as p ( r i | x, r 1 ..i − 1 , θ i ) and p ( r j | x, r 1 ..j − 1 , θ j ) where i ̸
Revised: Unlike a standard transformer decoder, the probabilities at different tree levels, such as p ( r i | x, r 1 ..i − 1 , θ i ) and p ( r j | x, r 1 ..j − 1 , θ j ) where i ̸

Original: = j , do not share parameters with each other.
Revised: = j , do not share parameters with each other.

Original: To distinguish different semantic levels, we concatenate the position and token values as input for each decoding step, as shown in the left corner of Figure 2.
Revised: To distinguish different semantic levels, we concatenate the position and token values as input for each decoding step, as shown in the left corner of Figure 2.

Original: Specifically, we have “ (1 , 3)(2 , 5)(3 , 5) ” for the semantic identifier “ 3 1 5 2 5 3 ”, while “ (2 , 5) ” and “ (3 , 5) ” represent different tokens in the vocabulary space.
Revised: Specifically, we have “ (1 , 3)(2 , 5)(3 , 5) ” for the semantic identifier “ 3 1 5 2 5 3 ”, while “ (2 , 5) ” and “ (3 , 5) ” represent different tokens in the vocabulary space.

Original: As the token embedding and linear classification layers share the same weights, the same token value in different positions would correspond to different model parameters.
Revised: As the token embedding and linear classification layers share the same weights, the same token value in different positions would correspond to different model parameters.

Original: Moreover, to reflect the influence of different prefixes, we expect the linear classification layer to be aware of different prefixes for predicting a specific token.
Revised: Moreover, to reflect the influence of different prefixes, we expect the linear classification layer to be aware of different prefixes for predicting a specific token.

Original: Concretely, instead of using the same projection weight W in the linear classification layer, we employ the prefix-aware adaptive weights for each token classifier, which can be calculated by another transformer decoder,
Revised: Concretely, instead of using the same projection weight W in the linear classification layer, we employ the prefix-aware adaptive weights for each token classifier, which can be calculated by another transformer decoder,

Original: [Equation]
Revised: [Equation]

Original: where e is the query embedding vector taken as initial input to the transformer decoder; { r t | t ∈ (1 , 2 , ..., i − 1) } are prefix tokens before the i -th position, AdaptiveDecoder stacks M 3 transformer decoding layers with dimension d , and W iada ∈
Revised: where e is the query embedding vector taken as initial input to the transformer decoder; { r t | t ∈ (1 , 2 , ..., i − 1) } are prefix tokens before the i -th position, AdaptiveDecoder stacks M 3 transformer decoding layers with dimension d , and W iada ∈

Original: R d × v is the adapted weight matrix for the corresponding classifier.
Revised: R d × v is the adapted weight matrix for the corresponding classifier.

Original: Finally, the i -th token in the given prefix can be predicted by Softmax ( h i W i ada ) .
Revised: Finally, the i -th token in the given prefix can be predicted by Softmax ( h i W i ada ) .

Original: For instance, to predict the third tokens in the identifiers “(1,3)(2,1)(3,5)” and “(1,2)(2,4)(3,5)”, respectively, the corresponding adaptive weights are derived separately for different prefixes, i.e. , “(1,3)(2,1)” and “(1,2)(2,4)”.
Revised: For instance, to predict the third tokens in the identifiers “(1,3)(2,1)(3,5)” and “(1,2)(2,4)(3,5)”, respectively, the corresponding adaptive weights are derived separately for different prefixes, i.e. , “(1,3)(2,1)” and “(1,2)(2,4)”.

Original: As we already know the previous tokens for each position in the teacher forcing setting, the prefix-aware adaptive weights can be calculated and trained in parallel in different positions while adding little burden to the entire model.
Revised: As we already know the previous tokens for each position in the teacher forcing setting, the prefix-aware adaptive weights can be calculated and trained in parallel in different positions while adding little burden to the entire model.

Original: 3.4 Training and inference
Revised: 3.4 Training and inference

Original: Consistency-based regularization.
Revised: Consistency-based regularization.

Original: To alleviate over-fitting, we employ a consistency-based regularization loss for training each decoding step.
Revised: To alleviate over-fitting, we employ a consistency-based regularization loss for training each decoding step.

Original: Given an input query q , we denote the decoder representations by two forward passes with independent dropouts before Softmax as z i, 1 = D ( r i | E ( q ) , r 1 ,...,i − 1 , θ i ) and z i, 2 = D
Revised: Given an input query q , we denote the decoder representations by two forward passes with independent dropouts before Softmax as z i, 1 = D ( r i | E ( q ) , r 1 ,...,i − 1 , θ i ) and z i, 2 = D

Original: ( r i | E ( q ) , r 1 ,...,i − 1 , θ i ) , respectively, where E ( · ) denotes the encoder network and D ( · ) denotes the decoder network.
Revised: ( r i | E ( q ) , r 1 ,...,i − 1 , θ i ) , respectively, where E ( · ) denotes the encoder network and D ( · ) denotes the decoder network.

Original: The consistency-based regularization loss tries to distinguish the representations from the same token from those of other tokens, like contrastive learning [8].
Revised: The consistency-based regularization loss tries to distinguish the representations from the same token from those of other tokens, like contrastive learning [8].

Original: The regularization loss of query q for the i -th decoding step is defined as,
Revised: The regularization loss of query q for the i -th decoding step is defined as,

Original: [Equation]
Revised: [Equation]

Original: where we leverage dot-product for sim ( · ) ; Q is the number of queries in the batch, and the temperature parameter is set as τ = 1 in all the experiments.
Revised: where we leverage dot-product for sim ( · ) ; Q is the number of queries in the batch, and the temperature parameter is set as τ = 1 in all the experiments.

Original: Training loss.
Revised: Training loss.

Original: Given a set of training examples D = { ( q, d ) } composed of queries (training queries and augmented queries) and document identifiers, the loss function can be written as follows:
Revised: Given a set of training examples D = { ( q, d ) } composed of queries (training queries and augmented queries) and document identifiers, the loss function can be written as follows:

Original: [Equation]
Revised: [Equation]

Original:  where p ( d | E ( q ) , θ ) denotes the probability of generating d with q as the input.
Revised: ) where p ( d | E ( q ) , θ ) denotes the probability of generating d with q as the input.

Original: The first part is the seq2seq cross-entropy loss with teacher forcing and the second part is the consistency-based regularization loss summed by all decoding steps.
Revised: The first part is the seq2seq cross-entropy loss with teacher forcing and the second part is the consistency-based regularization loss summed by all decoding steps.

Original: The whole process formulates a sequence-tosequence neural network, which can be optimized end-to-end via gradient descent.
Revised: The whole process formulates a sequence-tosequence neural network, which can be optimized end-to-end via gradient descent.

Original: The hyperparameter α denotes a scaling factor of regularization loss, which will be analyzed in Section 4.4.
Revised: The hyperparameter α denotes a scaling factor of regularization loss, which will be analyzed in Section 4.4.

Original: Inference via beam search.
Revised: Inference via beam search.

Original: In the inference stage, we calculate the query embedding through the encoder network and then perform beam search on the decoder network.
Revised: In the inference stage, we calculate the query embedding through the encoder network and then perform beam search on the decoder network.

Original: Due to the hierarchical nature of docid , it is convincing to constrain the beam search decoding process with a prefix tree, which in turn only generates the valid identifiers.
Revised: Due to the hierarchical nature of docid , it is convincing to constrain the beam search decoding process with a prefix tree, which in turn only generates the valid identifiers.

Original: The time complexity of beam search is O ( LBF ) , where L is the max length of identifiers (the depth of tree), B is the beam size and F is the max fanout of the tree (30 in our experiments).
Revised: The time complexity of beam search is O ( LBF ) , where L is the max length of identifiers (the depth of tree), B is the beam size and F is the max fanout of the tree (30 in our experiments).

Original: Given a balanced tree structure built by a corpus with N documents, the average time complexity for beam search is O ( B log N ) .
Revised: Given a balanced tree structure built by a corpus with N documents, the average time complexity for beam search is O ( B log N ) .

Original: We leave detailed descriptions of the constrained beam search algorithm in Appendix B.3.
Revised: We leave detailed descriptions of the constrained beam search algorithm in Appendix B.3.

Original:  Experiments
Revised:  Experiments

Original: In this section, we empirically verify the performance of NCI and the effectiveness of each component on the document retrieval task, which generates a ranking list of documents in response to a query.
Revised: In this section, we empirically verify the performance of NCI and the effectiveness of each component on the document retrieval task, which generates a ranking list of documents in response to a query.

Original: In the following, we discuss the datasets and evaluation protocols in Section 4.1, describe the implementation details and baseline methods in Section 4.2, and present empirical results and analyses in Section 4.3 and 4.4, respectively.
Revised: In the following, we discuss the datasets and evaluation protocols in Section 4.1, describe the implementation details and baseline methods in Section 4.2, and present empirical results and analyses in Section 4.3 and 4.4, respectively.

Original: 4.1 Datasets & evaluation metrics
Revised: 4.1 Datasets & evaluation metrics

Original: Datasets
Revised: Datasets.

Original: We conduct our experiments on two popular benchmarks for document retrieval, 
Revised: We conduct our experiments on two popular benchmarks for document retrieval, i.e. , the

Original: i. e , the Natural Questions 32] dataset andthe TriviaQA dataset 
Revised: Natural Questions [32] and TriviaQA dataset [29].

Original: • Natural Questions (NQ) [32] was introduced by Google in 2019.
Revised: Natural Questions (NQ) [32] was introduced by Google in 2019.

Original: The version we use is often referred to as NQ320 k , which consists of 320 k query-document pairs, where the documents are gathered from Wikipedia pages and the queries are natural language questions.
Revised: The version we use is often referred to as NQ320 k , which consists of 320 k query-document pairs, where the documents are gathered from Wikipedia pages and the queries are natural language questions.

Original: We use its predetermined training and validation split for evaluation.
Revised: We use its predetermined training and validation split for evaluation.

Original: • TriviaQA is a reading comprehension dataset [29], which includes 78 k query-document pairs from the Wikipedia domain.
Revised: TriviaQA is a reading comprehension dataset [29], which includes 78 k query-document pairs from the Wikipedia domain.

Original: Unlike the NQ320 k dataset, a query may include multiple answers in TriviaQA.
Revised: Unlike the NQ320 k dataset, a query may include multiple answers in TriviaQA.

Original: Metrics.
Revised: Metrics.

Original: We use widely accepted metrics for information retrieval, including Recall@ N , Mean
Revised: We use widely accepted metrics for information retrieval, including Recall@ N , Mean

Original: Reciprocal Rank (MRR) and R-precision.
Revised: Reciprocal Rank (MRR) and R-precision.

Original: Recall@ N measures how often the desired document is hit by the top- N retrieved candidates.
Revised: Recall@ N measures how often the desired document is hit by the top- N retrieved candidates.

Original: MRR calculates the reciprocal of the rank at which the first relevant document is retrieved.
Revised: MRR calculates the reciprocal of the rank at which the first relevant document is retrieved.

Original: R-Precision is the precision after R documents have been retrieved, where R is the number of relevant documents for the query.
Revised: R-Precision is the precision after R documents have been retrieved, where R is the number of relevant documents for the query.

Original: A high recall means that the ground truth document is contained in the retrieved candidate list, while a high MRR indicates that the corresponding document has already been ranked at the top position without re-ranking.
Revised: A high recall means that the ground truth document is contained in the retrieved candidate list, while a high MRR indicates that the corresponding document has already been ranked at the top position without re-ranking.

Original: 4.2 Implementation details
Revised: 4.2 Implementation details

Original: Hierarchical semantic identifier.
Revised: Hierarchical semantic identifier.

Original: For semantic identifiers, we apply a hierarchical k -means algorithm over the document embeddings obtained through a 12-layers BERT model with pre-trained parameters (provided by HuggingFace [53]).
Revised: For semantic identifiers, we apply a hierarchical k -means algorithm over the document embeddings obtained through a 12-layers BERT model with pre-trained parameters (provided by HuggingFace [53]).

Original: For each hierarchical layer, we employ the default k -means algorithm implemented in scikit-learn [42] with k = 30 .
Revised: For each hierarchical layer, we employ the default k -means algorithm implemented in scikit-learn [42] with k = 30 .

Original: For simplicity, the recursion terminal condition is also set as c = 30 .
Revised: For simplicity, the recursion terminal condition is also set as c = 30 .

Original: Query generation.
Revised: Query generation.

Original: We leverage the pre-trained model, DocT5Query
Revised: We leverage the pre-trained model, DocT5Query

Original: [40], for query generation.
Revised: [40], for query generation.

Original: We provide all document contents in NQ320 k and TriviaQA datasets to predict augmented querydocument pairs.
Revised: We provide all document contents in NQ320 k and TriviaQA datasets to predict augmented querydocument pairs.

Original: For each document, we generate 15 queries with the first 512 tokens of the document as input and constrain the maximum length of the generated query as 64.
Revised: For each document, we generate 15 queries with the first 512 tokens of the document as input and constrain the maximum length of the generated query as 64.

Original: Training and inference.
Revised: Training and inference.

Original: The Neural Corpus Indexer is implemented with python 3.6.10, PyTorch 1.8.1 and HuggingFace transformers 3.4.0.
Revised: The Neural Corpus Indexer is implemented with python 3.6.10, PyTorch 1.8.1 and HuggingFace transformers 3.4.0.

Original: We utilize the parameters of the T5 pre-trained model [5] to initialize the encoder and randomly initialize the PAWA decoder.
Revised: We utilize the parameters of the T5 pre-trained model [5] to initialize the encoder and randomly initialize the PAWA decoder.

Original: All NCI experiments are based on a learning rate 2 × 10 − 4 for the encoder and 1 × 10 − 4 for the decoder with a batch size 16 per
Revised: All NCI experiments are based on a learning rate 2 × 10 − 4 for the encoder and 1 × 10 − 4 for the decoder with a batch size 16 per

Original: GPU.
Revised: GPU.

Original: We set the scaling factor of the consistency-based regularization loss as α = 0 .
Revised: We set the scaling factor of the consistency-based regularization loss as α = 0 .

Original: 15 and the dropout ratio as 0 . 1 .
Revised: 15 and the dropout ratio as 0 . 1 .

Original: For inference, we apply the partial beam search algorithm to the trained seq2seq model.
Revised: For inference, we apply the partial beam search algorithm to the trained seq2seq model.

Original: We set the length penalty and the beam size as 0 . 8 and 100 , respectively.
Revised: We set the length penalty and the beam size as 0 . 8 and 100 , respectively.

Original: All experiments are based on a cluster of NVIDIA V100 GPUs with 32GB memory.
Revised: All experiments are based on a cluster of NVIDIA V100 GPUs with 32GB memory.

Original: Each job takes 8 GPUs, resulting in a total batch size of 128 ( 16 × 8 ).
Revised: Each job takes 8 GPUs, resulting in a total batch size of 128 ( 16 × 8 ).

Original: Baselines.
Revised: Baselines.

Original: We evaluate BM25 on both raw documents and those augmented by DocT5Query by an open-source implementation [2].
Revised: We evaluate BM25 on both raw documents and those augmented by DocT5Query by an open-source implementation [2].

Original: The performance of DSI [49] is referred from its original paper as the implementation has not been officially open-sourced.
Revised: The performance of DSI [49] is referred from its original paper as the implementation has not been officially open-sourced.

Original: To avoid the difference in data processing, we reproduce SEAL [4] and ANCE [54] by their official implementations.
Revised: To avoid the difference in data processing, we reproduce SEAL [4] and ANCE [54] by their official implementations.

Original: Some baselines for the
Revised: Some baselines for the

Original: TriviaQA dataset are directly referred from [58].
Revised: TriviaQA dataset are directly referred from [58].

Original: We leave the detailed settings in Appendix B.4.
Revised: We leave the detailed settings in Appendix B.4.

Original: 4.3 Results
Revised: 4.3 Results

Original: In Table 1 and 2, we compare the empirical results of NCI and corresponding baselines on two benchmarks.
Revised: In Table 1 and 2, we compare the empirical results of NCI and corresponding baselines on two benchmarks.

Original: We report NCI models based on T5-Base, T5-Large, and ensemble architectures.
Revised: We report NCI models based on T5-Base, T5-Large, and ensemble architectures.

Original: One can see that even with the T5-Base architecture, NCI outperforms all baselines by a significant margin across four different metrics on both the NQ320 k and TriviaQA datasets.
Revised: One can see that even with the T5-Base architecture, NCI outperforms all baselines by a significant margin across four different metrics on both the NQ320 k and TriviaQA datasets.

Original: Furthermore, an ensemble of five NCI models also brings a large enhancement, because each model is trained individually with a separate semantic identifier generated by a random k-means initialization, making the models complementary to each other.
Revised: Furthermore, an ensemble of five NCI models also brings a large enhancement, because each model is trained individually with a separate semantic identifier generated by a random k-means initialization, making the models complementary to each other.

Original: Expect for NCI, SEAL achieves the second best performance.
Revised: Expect for NCI, SEAL achieves the second best performance.

Original: This verifies the superiority of deep text retrieval over traditional sparse and dense retrieval methods.
Revised: This verifies the superiority of deep text retrieval over traditional sparse and dense retrieval methods.

Original: Comparing to SEAL, NCIfurther improves 17 .
Revised: Comparing to SEAL, NCI improves 17 .

Original: 6% for Recall@1, 10 . 0% for Recall@10, 3 . 2% for
Revised: 6% for Recall@1, 10 . 0% for Recall@10, 3 . 2% for

Original: Recall@100, and 14 . 9% for MRR@100 on the NQ320 k dataset.
Revised: Recall@100, and 14 . 9% for MRR@100 on the NQ320 k dataset.

Revised: We find that the generated queries have different distributions with the training queries , so we also fine-tune Doc2Query on this dataset for a comparison (denoted by w/ qg-ft ).

Revised: Finally, we achieve 72.78% for Recall@1, outperforming

Revised: SEAL by 21.4%.

Original: On the TriviaQA dataset, NCI obtains 7 .
Revised: On the TriviaQA dataset, NCI obtains 7 .

Original: 9% improvement for Recall@5, 5 . 5% for Recall@20, 6 . 0% for Recall@100, and 16 .
Revised: 9% improvement for Recall@5, 5 . 5% for Recall@20, 6 . 0% for Recall@100, and 16 .

Original: 8% for R-Precision.
Revised: 8% for R-Precision.

Original: As shown in ablation studies, these improvements are owning to the novel designs of PAWA decoder, query generation, semantic identifiers, and consistency-based regularization.
Revised: As shown in ablation studies, these improvements are owning to the novel designs of PAWA decoder, query generation, semantic identifiers, and consistency-based regularization.

Original: We also notice that query generation plays a key role in boosting the retrieval performance.
Revised: We also notice that query generation plays a key role in boosting the retrieval performance.

Original: With query generation, the BM25 + DocT5Query method achieves higher performance than the vanilla BM25, especially on the NQ320 k dataset.
Revised: With query generation, the BM25 + DocT5Query method achieves higher performance than the vanilla BM25, especially on the NQ320 k dataset.

Original: ANCE achieves competitive performance after fine-tuned by the training pairs, but the performance is relatively lower than our NCI model.
Revised: ANCE achieves competitive performance after fine-tuned by the training pairs, but the performance is relatively lower than our NCI model.

Original: Moreover, the MRR@ and R-Precision metrics of NCI are outstanding, indicating that more than 70% of the queries can be fulfilled without re-ranking on the retrieved document list.
Revised: Moreover, the MRR@100 and R-Precision metrics of NCI are outstanding, indicating that 80% of the queries can be fulfilled without re-ranking on the retrieved document list.

Original: This demonstrates the potential of NCI to be served as an end-to-end solution that replaces the entire index-retrieve-rank pipeline in traditional web search engines.
Revised: This demonstrates the potential of NCI to be served as an end-to-end solution that replaces the entire index-retrieve-rank pipeline in traditional web search engines.

Original: Furthermore, to study the effect of each component, we report ablation results on both NQ320 k and
Revised: Furthermore, to study the effect of each component, we report ablation results on both NQ320 k and

Original: TriviaQA datasets in Table 3.
Revised: TriviaQA datasets in Table 3.

Original: In general, all five components are able to improve the performance of document retrieval, which are detailed below.[Table] [Table]
Revised: In general, all five components are able to improve the performance of document retrieval, which are detailed below.

Original: w/o query generation.
Revised: w/o DocT5Query.

Original: This configuration removes the entire query generation module based on both DocT5Query and document as query components.
Revised: This configuration removes the training queries generated by DocT5Query .

Original: Remarkably, the query generation method boosts the performance greatly.
Revised: According to the results, the query generation model greatly boosts the performance. The result is [Table]

Original: The result is aligned with our expectation because training with the generated queries allows the  model to be agnostic to the semantic meanings of each document.
Revised: [Table] aligned with our expectation because training with augmented queries allows the NCI model to better understand the semantic meanings of each document [Table]

Original: As indicated by w/o DocT5Query and w/o document as query , both kinds of queries have significant impacts to  the final results
Revised: w/o document as query. Similar to DSI [49], using the document contents as queries also makes the model aware of the semantics of documents

Original: [Table]
Revised: [Table]

Original: w/o PAWA decoder.
Revised: w/o PAWA decoder.

Original: This configuration removes the adaptive decoder layer in Equation (4) and leverages shared weights with token embedding for the linear classification layer.
Revised: This configuration removes the adaptive decoder layer in Equation (4) and leverages shared weights with token embedding for the linear classification layer.

Original: We notice that the prefix-aware weight-adaptive decoder has a noticeable influence on the performance, which indicates that, instead of borrowing the vanilla transformer decoder, it is necessary to design a tailored decoder architecture for the task of semantic identifier generation.w/o semantic id. This configuration replaces the semantic identifier of each document to a randomly generated one.
Revised: We notice that the prefix-aware weight-adaptive decoder has a noticeable influence on the performance, which indicates that, instead of borrowing the vanilla transformer decoder, it is necessary to design a tailored decoder architecture for the task of semantic identifier generation.

Original: We find a relative drop in the model performance on all four metrics, demonstrating

Original: [Table]
Revised: [Table]

Original: 73.90 w/o query generation 46. 46.05 w/o DocT5Query 60.
Revised: [Figure]

Original: 63.50 w/o document as query 62. 67.48 w/o PAWA decoder 63. 71.81 w/o semantic id 62. 72.57 w/o regularization 65.
Revised: w/o semantic id. This configuration replaces the semantic identifier of each document to a randomly generated one.

Original: 71.59 w/o constrained beam search 65. [Figure] [Figure] that the semantic identifiers derived by the hierarchical k -means have injected useful priors.
Revised: We find a relative drop in the model performance on all four metrics, demonstrating that the semantic identifiers derived by the hierarchical k -means have injected useful priors.

Original: We conjecture that the performance enhancement would be more significant on a larger document corpus.
Revised: We conjecture that the performance enhancement would be more significant on a larger document corpus.

Original: w/o regularization.
Revised: w/o regularization.

Original: There is a performance drop on all four metrics without using consistency-based regularization loss.
Revised: There is a performance drop on all four metrics without using consistency-based regularization loss.

Original: The reason is that the decoder network is prone to over-fitting.
Revised: The reason is that the decoder network is prone to over-fitting.

Original: By making the prediction results of two augmented queries consistent, the decoder will become more generalizable and resistant to over-fitting.
Revised: By making the prediction results of two augmented queries consistent, the decoder will become more generalizable and resistant to over-fitting.

Original: w/o constrained beam search.
Revised: w/o constrained beam search.

Original: This configuration disables the validating constraint in beam search.
Revised: This configuration disables the validating constraint in beam search.

Original: In other words, the decoder network does not have a tree-based prior structure.
Revised: In other words, the decoder network does not have a tree-based prior structure.

Original: Instead, all tokens in the vocabulary can be generated in each decoding step.
Revised: Instead, all tokens in the vocabulary can be generated in each decoding step.

Original: We observe a performance drop on four evaluation metrics.
Revised: We observe a performance drop on four evaluation metrics.

Original: This indicates that it is difficult to remember all information of valid identifiers in the network, and an explicit prior could be helpful for improving the quality of beam search
Revised: This indicates that it is difficult to remember all information of valid identifiers in the network, and an explicit prior could be helpful for improving the quality of beam search.

Original: [Table][Table]
Revised: [Table]

Original: 4.4 Analysis
Revised: 4.4 Analysis

Original: Model capacity.
Revised: Model capacity.

Original: Figure 3 compares the learning curves of NCI with different model capacities, which are identical to the small, base, and large settings of ordinary T5 [43].
Revised: Figure 3 compares the learning curves of NCI with different model capacities, which are identical to the small, base, and large settings of ordinary T5 [43].

Original: We observe that with the increase of model size, NCI convergences more quickly with fewer epochs.
Revised: We observe that with the increase of model size, NCI convergences more quickly with fewer epochs.

Original: At convergence, the small model achieves a relatively lower recall.
Revised: At convergence, the small model achieves a relatively lower recall.

Original: Instead, both the base and large models achieve similar results after sufficient training epochs, and the large model will be slightly higher.
Revised: Instead, both the base and large models achieve similar results after sufficient training epochs, and the large model will be slightly higher.

Original: This implies that the model capacity has a critical impact on the retrieval performance, and the capacity of base model seems to be enough to memorize all documents in NQ320 k and TriviaQA datasets.
Revised: This implies that the model capacity has a critical impact on the retrieval performance, and the capacity of base model seems to be enough to memorize all documents in NQ320 k and TriviaQA datasets.

Original: The large model can be used when the computation capacity is sufficient.
Revised: The large model can be used when the computation capacity is sufficient.

Original: For a larger corpus, one may need to increase the model size to obtain satisfactory performance.
Revised: For a larger corpus, one may need to increase the model size to obtain satisfactory performance.

Original: Layer number of PAWA adapter.
Revised: Layer number of PAWA adapter.

Original: We study the influence of the number of transformer layers in the PAWA adapter and choose the layer number from {0,1,2,4,6,8}.
Revised: We study the influence of the number of transformer layers in the PAWA adapter and choose the layer number from {0,1,2,4,6,8}.

Original: The results are summarized in
Revised: The results are summarized in

Original: Table 4.
Revised: Table 4.

Original: We notice that with the increase of layer number, i.e. from 0 to 4, the overall performance is consistently improved on four metrics.
Revised: We notice that with the increase of layer number, i.e. from 0 to 4, the overall performance is consistently improved on four metrics.

Original: But when the number of layers achieves 6, the performance decreases.
Revised: But when the number of layers achieves 6, the performance decreases.

Original: When continuing to increase the number of layers to 8, the performance drops significantly.
Revised: When continuing to increase the number of layers to 8, the performance drops significantly.

Original: We attribute that to the overfitting issue caused by a large PAWA decoder.
Revised: We attribute that to the overfitting issue caused by a large PAWA decoder.

Original: Therefore, we adopt the
Revised: Therefore, we adopt the

Original: PAWA decoder with a 4-layers adapter in NCI.
Revised: PAWA decoder with a 4-layers adapter in NCI.

Original: Retrieved documents and their semantics identifiers.
Revised: Retrieved documents and their semantics identifiers.

Original: To verify the effectiveness of retrieval as well as the semantic identifiers learned by the hierarchical k -means, we analyze the retrieval results of NCI for some exemplar queries.
Revised: To verify the effectiveness of retrieval as well as the semantic identifiers learned by the hierarchical k -means, we analyze the retrieval results of NCI for some exemplar queries.

Original: To illustrate, we select four queries denoted by A-1 , A-2 , B-1 and
Revised: To illustrate, we select four queries denoted by A-1 , A-2 , B-1 and

Original: B-2 , where two queries inside the same group are semantically similar, and the queries in different groups correspond to distinct topics.
Revised: B-2 , where two queries inside the same group are semantically similar, and the queries in different groups correspond to distinct topics.

Original: In Figure 4, we show the probabilities of retrieved documents for each query in group A and B , respectively.
Revised: In Figure 4, we show the probabilities of retrieved documents [Figure] ili [Figure]

Revised: for each query in group A and B , respectively.

Original: The digits along x-axis denote the four-bit prefixes for the semantic identifiers of retrieved documents, and the y-axis stands for their probabilities.
Revised: The digits along x-axis denote the four-bit prefixes for the semantic identifiers of retrieved documents, and the y-axis stands for their probabilities.

Original: We notice that similar queries result in close document distributions, while dissimilar queries in different groups result in un-overlapped document collections.
Revised: We notice that similar queries result in close document distributions, while dissimilar queries in different groups result in un-overlapped document collections.

Original: In addition, the documents retrieved by the same group of queries have close prefixes for the identifiers, e.g. , 6030, 6032, 6033, 6034 in group
Revised: In addition, the documents retrieved by the same group of queries have close prefixes for the identifiers, e.g. , 6030, 6032, 6033, 6034 in group

Original: A and 7511, 7514, 7516 in group B .
Revised: A and 7511, 7514, 7516 in group B .

Original: Also, we visualize the BERT-based document embeddings by t-SNE [51] in Figure 4, in which each color represents the corresponding documents for a specific query.
Revised: Also, we visualize the BERT-based document embeddings by t-SNE [51] in Figure 4, in which each color represents the corresponding documents for a specific query.

Original: As shown in the figure, these documents naturally form two clusters with respect to different query groups.
Revised: As shown in the figure, these documents naturally form two clusters with respect to different query groups.

Original: Thus, we conclude that the semantic document identifiers generated by the hierarchical k -means algorithm have positive effects on the retrieval performance.
Revised: Thus, we conclude that the semantic document identifiers generated by the hierarchical k -means algorithm have positive effects on the retrieval performance.

Original: [Table]
Revised: [Table]

Original: Efficiency Analysis.
Revised: Efficiency Analysis.

Original: We use an NVIDIA V100-32G GPU to analyze the efficiency of NCI.
Revised: We use an NVIDIA V100-32G GPU to analyze the efficiency of NCI.

Original: As the inference speed is influenced by both model capacity and beam size, we report the latency and throughput measures for multiple settings in Table 5.
Revised: As the inference speed is influenced by both model capacity and beam size, we report the latency and throughput measures for multiple settings in Table 5.

Original: As NCI is an end-to-end retrieval method and achieves competitive performance without re-ranking, the latency and throughput are already affordable for some near-real-time applications.
Revised: As NCI is an end-to-end retrieval method and achieves competitive performance without re-ranking, the latency and throughput are already affordable for some near-real-time applications.

Original: The latency of NCI is on par with DSI and SEAL using the same model size and beam size, because all of them conduct beam search based on transformer decoders.
Revised: The latency of NCI is on par with DSI and SEAL using the same model size and beam size, because all of them conduct beam search based on transformer decoders.

Original: BM25 is very efficient (<100ms per query on CPU using an open-source implementation [2]), but the recall metrics are much lower.
Revised: BM25 is very efficient (<100ms per query on CPU using an open-source implementation [2]), but the recall metrics are much lower.

Original: Furthermore, we can leverage other techniques to improve the efficiency of NCI, which will be discussed in the later section.
Revised: Furthermore, we can leverage other techniques to improve the efficiency of NCI, which will be discussed in the later section.

Original:  Limitation & Future Works
Revised:  Limitation & Future Works

Original: Despite the significant breakthrough, the current implementation of NCI still suffers from several limitations before deployment in a large-scale search system.
Revised: Despite the significant breakthrough, the current implementation of NCI still suffers from several limitations before deployment in a large-scale search system.

Original: Firstly, it requires a much larger model capacity for extending NCI to the web scale.
Revised: Firstly, it requires a much larger model capacity for extending NCI to the web scale.

Original: Secondly, the inference speed needs to be improved to serve online queries in real time.
Revised: Secondly, the inference speed needs to be improved to serve online queries in real time.

Original: Thirdly, it is difficult to update the model-based index when new documents are added to the system.
Revised: Thirdly, it is difficult to update the model-based index when new documents are added to the system.

Original: In future works, we may tackle these problems from four aspects.
Revised: In future works, we may tackle these problems from four aspects.

Original: (1) The architecture of sparsely-gated Mixture of Expert (MoE) [47] can be employed to enhance the model capacity.
Revised: (1) The architecture of sparsely-gated Mixture of Expert (MoE) [47] can be employed to enhance the model capacity.

Original: (2) Documents can be grouped into semantic clusters, and NCI can be used to retrieve relevant cluster identifiers.
Revised: (2) Documents can be grouped into semantic clusters, and NCI can be used to retrieve relevant cluster identifiers.

Original: In this way, all documents in relevant clusters can be retrieved efficiently.
Revised: In this way, all documents in relevant clusters can be retrieved efficiently.

Original: (3) Model compression techniques, like weight quantization [26] and knowledge distillation [24], can be further taken to speed up inference.
Revised: (3) Model compression techniques, like weight quantization [26] and knowledge distillation [24], can be further taken to speed up inference.

Original: (4) We plan to explore a hybrid solution by building another index that serves new documents through traditional indexing algorithms.
Revised: (4) We plan to explore a hybrid solution by building another index that serves new documents through traditional indexing algorithms.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we introduce a novel document retrieval paradigm that unifies the training and indexing stages by an end-to-end deep neural network.
Revised: In this work, we introduce a novel document retrieval paradigm that unifies the training and indexing stages by an end-to-end deep neural network.

Original: The proposed Neural Corpus Indexer (NCI) directly retrieves the identifiers of relevant documents for an input query, which can be optimized end-to-end using augmented query-document pairs.
Revised: The proposed Neural Corpus Indexer (NCI) directly retrieves the identifiers of relevant documents for an input query, which can be optimized end-to-end using augmented query-document pairs.

Original: To optimize the recall and ranking performance, we invent a tailored prefix-aware weight-adaptive decoder.
Revised: To optimize the recall and ranking performance, we invent a tailored prefix-aware weight-adaptive decoder.

Original: Empirically, we evaluate NCI on NQ320 k and
Revised: Empirically, we evaluate NCI on NQ320 k and

Original: TriviaQA datasets, demonstrating its outstanding performance over state-of-the-art solutions.
Revised: TriviaQA datasets, demonstrating its outstanding performance over state-of-the-art solutions.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section.
Revised: [Yes] See Section.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See appendix.
Revised: [Yes] See appendix.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See appendix.
Revised: [Yes] See appendix.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] 5.
Revised: [Yes] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Related work
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Traditional web search techniques follow a two-stages paradigm including document retrieval and document ranking .

Original: The first stage aims to select a collection of documents relevant to a given query, which requires an ingenious trade-off between efficiency and recall.

Original: Then, the document ranking stage takes more advanced features and deeper models to calculate a fine-grained ranking score for each query and document pair.

Original: In the following, we first discuss related works for document retrieval and ranking respectively.

Original: Afterwards, we introduce recent works that incorporate pre-trained language models into these two stages.

Original: At last, the attempts on end-to-end retrieval will be discussed.

Original: A.1 Document retrieval

Original: Traditional document retrieval methods are based on Sparse Retrieval , which is built upon inverted index with term matching metrics such as TF-IDF [45], query likelihood [33] or BM25 [44].

Original: In industry-scale web search, BM25 is a difficult-to-beat baseline owing to its outstanding trade-off between accuracy and efficiency.

Original: In recent years, there are some attempts to incorporate the power of neural networks into inverted index.

Original: The Standalone Neural Ranking Model (SNRM) [57] learns high-dimensional sparse representations for queries and documents, which enables the construction of inverted index for efficient document retrieval.

Original: Doc2Query [41] predicts relevant queries to augment the content of each document before building the BM25 index, and DocT5Query [40] improves the performance of query generation by the pre-trained language model T5 [5].

Original: Furthermore, DeepCT [9] calculates context-aware term importance through neural networks to improve the term matching metrics of BM25.

Original: Another line of research lies in Dense Retrieval , which presents query and documents in dense vectors and models their similarities with inner product or cosine similarity.

Original: These methods benefit from recent progresses of pre-trained language models, such as BERT [14] and RoBERTa [35] to obtain dense representations for queries and documents.

Original: At inference time, efficient Approximate Nearest

Original: Neighbor (ANN) search algorithms, such as k-dimensional trees [3], locality-sensitive hashing [10], and graph-based indexes (e.g., HNSW [38], DiskANN [27] and SPANN [7]) can be utilized to retrieve relevant documents within a sublinear time.

Original: Besides, Luan et al.

Original: [37] analyze the limited capacity of dual encoders, and propose a combination of sparse and dense retrieval methods with multi-vector encoding to achieve better search quality.

Original: A.2 Document ranking

Original: Document ranking has been extensively studied in recent years and experienced a huge improvement with the booming of deep neural networks.

Original: Neural network-based document ranking models mainly fall into two categories.

Original: Representation-based models like DSSM (Deep Structured Semantic

Original: Model) [25] and CDSSM (a convolution-based variant of DSSM)

Original: [48] represent query and document in a shared semantic space and model their semantic similarity through a neural network.

Original: In contrast,

Original: Interaction-based models first build interactions between query and document terms, and then utilizes neural networks to learn hierarchical interaction patterns.

Original: For example, DRMM (Deep Relevance

Original: Matching Model) [21] extracts interactive features by matching histograms and utilizing a feed forward network with term-gating mechanism to calculate the relevance score of a query-document pair.

Original: A.3 Pre-trained language models

Original: Recently, Pre-trained Language Models (PLMs) like BERT [14] have led to a revolution of web search techniques.

Original: The vanilla BERT model utilizes a single-tower architecture that concatenates query and document tokens as a whole input to the relevance model.

Original: Despite of its superior performance, the high computational cost hinders its application to industrial-scale web search systems.

Original: TwinBERT [36] tackles this problem by exploiting a Siamese architecture, where queries and documents are first modeled by two BERT encoders separately, and then an efficient crossing layer is adopted for relevance calculation.

Original: The representation vectors for all documents can be calculated and indexed offline.

Original: In the online serving stage, it calculates the representation vector for the input query and applies a crossing layer to calculate the relevance score between each query and document.

Original: The crossing layer usually adopts simple similarity functions such as dot product or a single feed-forward layer to achieve a high efficiency.

Original: Moreover, Chang et al.

Original: [6] argue that the Masked Language Model (MLM) loss designed for BERT pre-training is not naturally fitted to embedding-based retrieval tasks.

Original: Instead, they propose three paragraph-level pre-training tasks, i.e. , Inverse Cloze Task (ICT), Body First Selection (BFS), and

Original: Wiki Link Prediction (WLP), which demonstrate promising results in text retrieval experiments.

Original: Gao et al.

Original: [16] find that a standard LMs’ internal attention structure is not ready-to-use for dense encoders.

Original: Thus, they propose a novel architecture named Condenser to improve the performance of dense retrieval.

Original: ANCE (Approximate nearest neighbor Negative Contrastive Estimation) [54] leverages hard negatives to improve the effectiveness of contrastive learning, which generates better text representations for the retrieval task.

Original: A.4 End-to-end retrieval

Original: The deficiency of index-retrieve paradigm lies in that the two stages of document retrieval and re-ranking are optimized separately.

Original: Especially, the document retrieval procedure is often sub-optimal and hinders the performance of the entire system.

Original: Thus, there are some recent attempts to achieve endto-end retrieval as a one-stage solution.

Original: ColBERT [31] introduces a contextualized late interaction architecture, which independently encodes query and document through BERT, and performs crossterm interaction based on the contextualized representations of query and document terms.

Original: ColBERT supports end-to-end retrieval directly from a large document collection by leveraging vector-similarity indexes in the pruned interaction layer.

Original: It can be viewed as a compromise between single-tower and twin-tower BERT architectures which maintains an effective trade-off between accuracy and latency.

Original: Moreover, the Contextualized Inverted List (COIL) [19] exacts lexical patterns from exact matching pairs through contextualized language representations.

Original: At search time, we build representation vectors for query tokens and perform contextualized exact match to retrieve relevant documents based on inverted index.

Original: Although ColBERT and COIL have shown promising results in end-to-end retrieval tasks without re-ranking, their performance is still not obviously better (if not worse) than a common practice of “BM25 indexer + BERT re-ranker”, and their efficiency is also not good enough for an industrial web search engine.

Original: Therefore, we resort to a new indexing paradigm to break the bottleneck.

Original: We believe the neural corpus indexer proposed in this paper is a crucial break-through, opening up new opportunities to optimize the performance of web-scale document retrieval.

Original: Moreover, there are a few attempts that try to build a model-based search index by directly predicting document identifiers.

Original: Tay et al.

Original: [50] proposed the DSI (differentiable search index) model based on an encoder-decoder architecture to generate relevant docids.

Original: However, its decoder architecture remains the same as T5, which is unsuitable to generate semantic ids derived by hierarchical k -means.

Original: SEAL [4] uses all n-grams in a passage as its possible identifiers and build a FM-Index to retrieve documents; but it is hard to enumerate all n-grams for retrieving relevant documents.

Original: In addition, our work is related to Deep Retrieval [20] for the recommendation task, which learns a deep retrievable network with user-item clicks without resorting to ANN algorithms constrained by the Euclidean space assumption.

Original: B Reproducibility

Original: We provide our code for reproduction in the supplementary material.

Original: We will release it to public shortly.

Original: B.1 Dataset processing

Original: We conduct experiments on NQ320 k and TriviaQA datasets.

Original: For NQ320 k dataset, the queries are natural language questions and the documents are Wikipedia articles in HTML format.

Original: During dataset processing, we first filter out useless HTML tag tokens, and extract title, abstract and content strings of each Wikipedia article using regular expression.

Original: The experiments are also conducted on TriviaQA dataset.

Original: For TriviaQA dataset, it includes 78 k query-document pairs from the Wikipedia domain, which are processed almost the same as NQ320 k .

Original: Then, we detect duplicated articles based on the title of each article.

Original: After that, we concatenate the title, abstract and content strings of each Wikipedia article, and apply a 12 layers pre-trained BERT model on it to generate document embeddings.

Original: Finally, hierarchical k -means is applied on the article embeddings to produce semantic identifiers for each article.

Original: B.2 Hierarchical k -means for semantic identifier

Original: The pseudo code of hierarchical k -means is detailed in in Algorithm 1.

Original: Algorithm

Original: Input:

Original: Document embedding X 1: N Number of clusters k

Original: Recursion terminal condition c

Original: Output:

Original: Hierarchical semantic identifier L 1: N Function:

Original: GenerateSemanticIdentifier ( X 1: N ) C 1: k ← KMeansCluster ( X 1:

Original: N , k ) L ← ∅ for i ∈ [0, k − 1 ] do

Original: L current ← [ i ] ∗ | C

Original: [-i +1 | if | C i +
-]
Revised: [-i +1 | if | C i +
-]

Original: | >

Original: c then

Original: L rest ← GenerateSemanticIdentifier ( C i +1 ) else

Original: L rest ← [0 , ..., | C i +1 | − 1] end if

Original: L cluster ← ConcatString( L current , L rest )

Original: L ← L .Append( L cluster ) end for

Original: L ← reorderToOriginal( L, X 1: N , C 1: k ) return L

Original: B.3 Constrained beam search

Original: The pseudo code of constrained beam search is detailed in Algorithm 2.

Original: B.4 Baselines

Original: We describe the baseline methods in this section.

Original: For most of them, we use their official open-source implementations

Original: • BM25.

Original: BM25 is currently the mainstream algorithm for calculating the similarity score between query and document in information retrieval [44].

Original: We calculate BM25 between an original query Q and a document d which derived from a sum of contributions from each query term q i as,

Original: Score( Q, d ) =

Original: [Equation]

Original: i

Original: [Equation]

Original: [Equation]

Original: where w i denotes the weight of q i , and R ( q i , d ) is the correlation between q i and d .

Original: We use the open-source implementation from Rank-BM25 2 .

Original: • BM25 + DocT5Query.

Original: The docT5Query model [40] generates questions that related to a document.

Original: These predicted queries are then appended to the original documents, which are then

Original: indexed.

Original: that we use the same predicted queries in our query generation module.

Original: Queries are issued against the index as “bag of words” queries, using BM25 for evaluation.

Original: We the

Original: -source code for

Original: 3 , and the generated queries keep the with NCI (our model) to have a fair comparison

Original: Input:

Original: Query embedding x

Original: Beam search size k

Original: Max beam length n

Original: Prefix tree T with root r 0 , containing all valid identifiers

Original: Log probability function f ( r i )

Original: =

Original: log ( p ( r i | x, r i − 1 , ..., r 0 ))

Original: Output: k documents with the highest probabilities

Original: Function: prefix ={ r 0 }

Original: ResultIds ← ∅

Original: B 0 ← {⟨ prefix , EOS ⟩} for i ∈ [1, n ] do for ⟨ prefix , sum _ log _ prob ⟩ ∈ B i − 1 do if prefix .last().isLeaf

Original: () then doc _ id = prefix .toString

Original: ResultIds .add( ⟨ doc _ id, sum _ log _ prob/len ( prefix ) ⟩ ) elsefor r i ∈ prefix .last().child

Original: () do new _ prefix = prefix .copy().append

Original: ( r i )

Original: B i .add

Original: ( ⟨ new _ prefix , sum _ log _ prob + f ( r i ) ⟩ ) end for end if end for

Original: B

Original: i ← B

Original: i .rank_by_prob().top( k ) end for return ResultIds .rank_by_prob().top( k )

Original: • BERT + ANN (Faiss).

Original: We use the Flat Index method with the query and document representations obtained by CoCondenser 4 [16] which is pretrained on Wikipedia and then finetuned over NQ dataset.

Original: For the Flat Index method, we use the version implemented by Faiss 5 .

Original: • BERT + BruteForce.

Original: In this baseline, we use the CoCondenser [16], pretrained on Wikipedia and then finetuned over NQ dataset, to encode queries and documents separately.

Original: Then, the Cosine Similarity is computed for each query and document pair.

Original: After that, for each query, the documents with the largest Cosine Similarity score are retrieved.

Original: • ANCE (MaxP & FirstP).

Original: ANCE, a training mechanism, that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus [54].

Original: For BERT FirstP, we concatenate the title and content of each document by a [SEP] token.

Original: For BERT MaxP, we only use the content of each document.

Original: We use the open-source implementation 6 .

Original: • SEAL (BART-Large).

Original: We reproduce SEAL based on the open-sourced implementation 7 .

Original: • DSI.

Original: The DSI model learns a text-to-text model that maps string queries directly to relevant docids [50].

Original: We report the performance of DSI (T5-Base), DSI (T5-Large) and DSI (T5-XXL) from its original paper as the implementation has not been open-sourced.

Original: C More Experimental Results

Original: We study the influence of regularization strength and choose the regularization hyper-parameter α from {0, 0.1, 0.15, 0.2, 0.3}.

Original: Table 6 summaries the results with different regularization hyperparameter α settings.

Original: At convergence, the hyper-parameter α = 0.15 generally achieves better performance.

Original: Therefore, we set the default value as α = 0.15 in NCI.

Original: [Table]
Revised: [N/A]

Original: D Miscellaneous

Original: Social Impacts.

Original: This work aims at introducing a new learning paradigm that can unify the learning and indexing stages with an end-to-end deep neural network.

Original: Besides, our work has the potential to inspire more attempts at unifying the retrieval and re-ranking task with an end-to-end framework, which might have positive social impacts.

Original: We do not foresee any form of negative social impact induced by our work.

Original: Privacy Information in Data.

Original: We use the NQ dataset privided by the work [32].

Original: The dataset only includes questions, rendered Wikipedia pages, tokenized representations of each page, and the annotations added by our annotators.

Original: No privacy information is included.

Original: For the TriviaQA [29], which is a reading comprehension dataset, it includes 78 k query-document pairs from the Wikipedia domain.

Original: Again, no privacy information is included.