Original: Autoregressive Search Engines: Generating Substrings as Document Identifiers
Revised: Autoregressive Search Engines: Generating Substrings as Document Identifiers

Original: Abstract
Revised: Abstract

Original: Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus.
Revised: Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus.

Original: Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at an astonishing pace.
Revised: Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at an astonishing pace.

Original: In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models’ architecture.
Revised: In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models’ architecture.

Original: Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier.
Revised: Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier.

Original: In this work we propose an alternative that doesn’t force any structure in the search space: using all ngrams in a passage as its possible identifiers.
Revised: In this work we propose an alternative that doesn’t force any structure in the search space: using all ngrams in a passage as its possible identifiers.

Original: This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through an efficient data structure.
Revised: This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through an efficient data structure.

Original: Empirically, we show this not only outperforms prior autoregressive approaches but also leads to an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new stateof-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems.
Revised: Empirically, we show this not only outperforms prior autoregressive approaches but also leads to an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new stateof-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems.

Original: Code and pre-trained models are available at https://github.com/facebookresearch/SEAL .
Revised: Code and pre-trained models are available at https://github.com/facebookresearch/SEAL .

Original:  Introduction
Revised:  Introduction

Original: Surfacing knowledge from large corpora is a crucial step when dealing with knowledge intensive language tasks [Levy et al., 2017, Dinan et al., 2019, Elsahar et al., 2018, Petroni et al., 2021], such as open-domain question answering [Voorhees et al., 1999, Joshi et al., 2017, Yang et al., 2018,
Revised: Surfacing knowledge from large corpora is a crucial step when dealing with knowledge intensive language tasks [Levy et al., 2017, Dinan et al., 2019, Elsahar et al., 2018, Petroni et al., 2021], such as open-domain question answering [Voorhees et al., 1999, Joshi et al., 2017, Yang et al., 2018,

Original: Kwiatkowski et al., 2019] and fact checking [Thorne et al., 2018].
Revised: Kwiatkowski et al., 2019] and fact checking [Thorne et al., 2018].

Original: A popular paradigm to approach such tasks is to combine a search engine with a machine reader component.
Revised: A popular paradigm to approach such tasks is to combine a search engine with a machine reader component.

Original: The former retrieves relevant context, usually in the form of short passages, which the latter then examines to produce answers [Chen et al., 2017, Lewis et al., 2020, Izacard and Grave, 2021].
Revised: The former retrieves relevant context, usually in the form of short passages, which the latter then examines to produce answers [Chen et al., 2017, Lewis et al., 2020, Izacard and Grave, 2021].

Original: In recent years we have witnessed a surge of research and development in autoregressive language models [Radford et al., 2019, Lewis et al., 2019, Raffel et al., 2019, Brown et al., 2020, Rae et al., 2021,
Revised: In recent years we have witnessed a surge of research and development in autoregressive language models [Radford et al., 2019, Lewis et al., 2019, Raffel et al., 2019, Brown et al., 2020, Rae et al., 2021,

Original: Artetxe et al., 2021, Smith et al., 2022], with ever increasing size and natural language understanding (NLU) capabilities.
Revised: Artetxe et al., 2021, Smith et al., 2022], with ever increasing size and natural language understanding (NLU) capabilities.

Original: Such models are currently the de-facto implementation of the machine reader component in retrieval-reader architectures, and have contributed to rapid progress on a wide range of benchmarks [Joshi et al., 2017, Kwiatkowski et al., 2019, Petroni et al., 2021].
Revised: Such models are currently the de-facto implementation of the machine reader component in retrieval-reader architectures, and have contributed to rapid progress on a wide range of benchmarks [Joshi et al., 2017, Kwiatkowski et al., 2019, Petroni et al., 2021].

Original: However, these tremendous advances in aggressive modelling has yet to bring similar transformational changes in how retrieval is approached.
Revised: However, these tremendous advances in aggressive modelling has yet to bring similar transformational changes in how retrieval is approached.

Original: Transferring the NLU capabilities of modern autoregressive models to retrieval is non-trivial.
Revised: Transferring the NLU capabilities of modern autoregressive models to retrieval is non-trivial.

Original: Some works have demonstrated that knowledge stored in the parameters of these models can be retrieved to some extend by directly generating evidence given a query [Petroni et al., 2019, 2020, Roberts
Revised: Some works have demonstrated that knowledge stored in the parameters of these models can be retrieved to some extend by directly generating evidence given a query [Petroni et al., 2019, 2020, Roberts

Original: Carbon footprint
Revised: Carbon footprint

Original: Carbon dioxide is released naturally by decomposition, ocean release and respiration.
Revised: Carbon dioxide is released naturally by decomposition, ocean release and respiration.

Original: Humans contribute an increase of carbon dioxide emissions by burning fossil fuels, deforestation, and cement production.
Revised: Humans contribute an increase of carbon dioxide emissions by burning fossil fuels, deforestation, and cement production.

Original: Methane (CH4) is largely released by coal, oil, and natural gas industries.
Revised: Methane (CH4) is largely released by coal, oil, and natural gas industries.

Original: Although methane is not mass-produced like carbon dioxide, it is still very prevalent.
Revised: Although methane is not mass-produced like carbon dioxide, it is still very prevalent.

Original: Autoregressive language model Query:  Causes of co2 increase
Revised: Autoregressive language model Query:  Causes of co2 increase

Original: [Figure]
Revised: [Figure]

Original: which
Revised: which

Original: [Figure]
Revised: [Figure]

Original: SEAL et al., 2020].
Revised: SEAL et al., 2020].

Original: However, such approaches have been shown to be unreliable because of their tendency to hallucinate non-factual content [Massarelli et al., 2019, Metzler et al., 2021, Ji et al., 2022].
Revised: However, such approaches have been shown to be unreliable because of their tendency to hallucinate non-factual content [Massarelli et al., 2019, Metzler et al., 2021, Ji et al., 2022].

Original: To alleviate this issue, previous work proposes to only use generation for query expansion in traditional search engines [Mao et al., 2021], but these solutions don’t exploit the full potential of autoregressive architecture, such as word order sensitivity and conditional probability modeling, and still lag behind vector-based approaches [Karpukhin et al., 2020].
Revised: To alleviate this issue, previous work proposes to only use generation for query expansion in traditional search engines [Mao et al., 2021], but these solutions don’t exploit the full potential of autoregressive architecture, such as word order sensitivity and conditional probability modeling, and still lag behind vector-based approaches [Karpukhin et al., 2020].

Original: Recently, another line of work has investigated using autoregressive language models to generate identifier strings for documents as an intermediate target for retrieval, such as Wikipedia page titles [De Cao et al., 2021b], or root-to-leaf paths in a hierarchical cluster tree [Tay et al., 2022].
Revised: Recently, another line of work has investigated using autoregressive language models to generate identifier strings for documents as an intermediate target for retrieval, such as Wikipedia page titles [De Cao et al., 2021b], or root-to-leaf paths in a hierarchical cluster tree [Tay et al., 2022].

Original: Employing identifiers, rather than generating evidence directly, induces structure in the search space, ( i.e. , index documents by their title or their cluster tree) which can be easier to memorize, learn, and retrieve from, than full unstructured passages.
Revised: Employing identifiers, rather than generating evidence directly, induces structure in the search space, ( i.e. , index documents by their title or their cluster tree) which can be easier to memorize, learn, and retrieve from, than full unstructured passages.

Original: Moreover, it is relatively easy to constrain decoding with a prefix tree “index” so that only valid identifiers are generated.
Revised: Moreover, it is relatively easy to constrain decoding with a prefix tree “index” so that only valid identifiers are generated.

Original: As a downside, if appropriate metadata ( e.g. , titles) are not available, one needs to create the identifiers, hence the structure ( e.g. , with hierarchical clustering), which has not been thoroughly evaluated on a large-scale benchmark.
Revised: As a downside, if appropriate metadata ( e.g. , titles) are not available, one needs to create the identifiers, hence the structure ( e.g. , with hierarchical clustering), which has not been thoroughly evaluated on a large-scale benchmark.

Original: In this work, we propose a solution that does not force any structure in the search space, but rather uses all the ngrams occurring in a document as its identifiers.
Revised: In this work, we propose a solution that does not force any structure in the search space, but rather uses all the ngrams occurring in a document as its identifiers.

Original: Concretely, we introduce S earch E ngines with A utoregressive L Ms (SEAL), a retrieval solution that combines an autoregressive model, i.e. ,
Revised: Concretely, we introduce S earch E ngines with A utoregressive L Ms (SEAL), a retrieval solution that combines an autoregressive model, i.e. ,

Original: BART [Lewis et al., 2019], with a compressed full-text substring index, i.e. , the FM-Index [Ferragina and Manzini, 2000] — see Figure 1 for an high-level overview.
Revised: BART [Lewis et al., 2019], with a compressed full-text substring index, i.e. , the FM-Index [Ferragina and Manzini, 2000] — see Figure 1 for an high-level overview.

Original: This configuration comes with a twofold benefit: i) we can constrain BART’s generations with the FM-Index, hence preventing the generation of invalid identifiers ( i.e. , ngrams not occurring in any document); ii) the FM-Index provides information on all documents in the corpus containing a specific ngram (for every decoding step), thus allowing to retrieve them.
Revised: This configuration comes with a twofold benefit: i) we can constrain BART’s generations with the FM-Index, hence preventing the generation of invalid identifiers ( i.e. , ngrams not occurring in any document); ii) the FM-Index provides information on all documents in the corpus containing a specific ngram (for every decoding step), thus allowing to retrieve them.

Original: This setup allows SEAL to generate any span from any position in the corpus, without needing to explicitly encode all substrings in a document.
Revised: This setup allows SEAL to generate any span from any position in the corpus, without needing to explicitly encode all substrings in a document.

Original: Moreover, we design a novel scoring function to intersect the results of multiple ngrams combining LM probabilities with
Revised: Moreover, we design a novel scoring function to intersect the results of multiple ngrams combining LM probabilities with

Original: FM-index frequencies ( i.e. , number of occurrences of the ngram in the whole corpus).
Revised: FM-index frequencies ( i.e. , number of occurrences of the ngram in the whole corpus).

Original: Our experimental evaluation shows that SEAL matches or outperforms recent retrieval solutions (including autoregressive ones) on Natural Questions
Revised: Our experimental evaluation shows that SEAL matches or outperforms recent retrieval solutions (including autoregressive ones) on Natural Questions

Original: [Kwiatkowski et al., 2019], while requiring substantially less memory ( ∼ 2 to 7 times smaller in footprint).
Revised: [Kwiatkowski et al., 2019], while requiring substantially less memory ( ∼ 2 to 7 times smaller in footprint).

Original: Moreover, SEAL’s intersection formulation improves the state-of-the-art on passage-level retrieval by more than 10 points on the
Revised: Moreover, SEAL’s intersection formulation improves the state-of-the-art on passage-level retrieval by more than 10 points on the

Original: KILT benchmark [Petroni et al., 2021], contributing in establishing new state-of-the-art downstream results on multiple datasets when paired with existing reader technologies.
Revised: KILT benchmark [Petroni et al., 2021], contributing in establishing new state-of-the-art downstream results on multiple datasets when paired with existing reader technologies.

Original:  Related Work
Revised:  Related Work

Original: One way to approach retrieval with autoregressive models makes use of unique identifiers, i.e. , string pointers to documents that are in some way easier to generate than the full document itself [De Cao et al., 2021b, Tay et al., 2022].
Revised: One way to approach retrieval with autoregressive models makes use of unique identifiers, i.e. , string pointers to documents that are in some way easier to generate than the full document itself [De Cao et al., 2021b, Tay et al., 2022].

Original: In our work the identifiers are generated ngrams, which do not necessarily occur in just one document.
Revised: In our work the identifiers are generated ngrams, which do not necessarily occur in just one document.

Original: Note that the idea of using phrases to retrieve passages has shown promise already in the context of dense retrieval [Lee et al., 2021a,b].
Revised: Note that the idea of using phrases to retrieve passages has shown promise already in the context of dense retrieval [Lee et al., 2021a,b].

Original: Our approach is quite conceptually distinct from document [Nogueira et al., 2019, Nogueira and
Revised: Our approach is quite conceptually distinct from document [Nogueira et al., 2019, Nogueira and

Original: Lin, 2021] and query expansion approaches [Mao et al., 2021], that use autoregressive models to boost search, but still rely on black-box systems like BM25 [Robertson and Zaragoza, 2009] for the retrieval: in our work the boundary between generation and retrieval is blurred, since we generate grounded passage spans.
Revised: Lin, 2021] and query expansion approaches [Mao et al., 2021], that use autoregressive models to boost search, but still rely on black-box systems like BM25 [Robertson and Zaragoza, 2009] for the retrieval: in our work the boundary between generation and retrieval is blurred, since we generate grounded passage spans.

Original: The same reasoning applies to models that generate queries for web search engines [Komeili et al., 2021, Shuster et al., 2022, Nakano et al., 2021, Lazaridou et al., 2022].
Revised: The same reasoning applies to models that generate queries for web search engines [Komeili et al., 2021, Shuster et al., 2022, Nakano et al., 2021, Lazaridou et al., 2022].

Original: Virtually all modern approaches to string-matching-based retrieval use bag-of-words representations.
Revised: Virtually all modern approaches to string-matching-based retrieval use bag-of-words representations.

Original: Many recent works propose learned contextualized weighting for both queries and documents terms [Dai and Callan, 2019, Gao et al., 2021, Lin and Ma, 2021, Mallia et al., 2021, Dai and Callan, 2020, Bai et al., 2020, Zhao et al., 2021, Formal et al., 2021b,a].
Revised: Many recent works propose learned contextualized weighting for both queries and documents terms [Dai and Callan, 2019, Gao et al., 2021, Lin and Ma, 2021, Mallia et al., 2021, Dai and Callan, 2020, Bai et al., 2020, Zhao et al., 2021, Formal et al., 2021b,a].

Original: Many of these methods can also weigh terms that are not present in the query, addressing so-called vocabulary mismatch.
Revised: Many of these methods can also weigh terms that are not present in the query, addressing so-called vocabulary mismatch.

Original: In contrast,
Revised: In contrast,

Original: SEAL generates (and scores) ngrams of arbitrary size.
Revised: SEAL generates (and scores) ngrams of arbitrary size.

Original: These approaches are partly orthogonal to
Revised: These approaches are partly orthogonal to

Original: SEAL, as many of the proposed techniques could be used to rescore higher-order ngrams.
Revised: SEAL, as many of the proposed techniques could be used to rescore higher-order ngrams.

Original: Finally, a connected strand of research is that of query likelihood models, which use autoregressive models to (re)rank passages according to the probability P ( q | p ) of a query q given the passage p
Revised: Finally, a connected strand of research is that of query likelihood models, which use autoregressive models to (re)rank passages according to the probability P ( q | p ) of a query q given the passage p

Original: [Nogueira dos Santos et al., 2020, Zhuang and Zuccon, 2021, Lesota et al., 2021, Sachan et al., 2022].
Revised: [Nogueira dos Santos et al., 2020, Zhuang and Zuccon, 2021, Lesota et al., 2021, Sachan et al., 2022].

Original: In our case, the autoregressive architecture models the likelihood of an ngram given the query, i.e. , P ( n | q ) .
Revised: In our case, the autoregressive architecture models the likelihood of an ngram given the query, i.e. , P ( n | q ) .

Original:  Background
Revised:  Background

Original: In retrieval, the automatic system is required to return an ordered list of documents d 1 , d 2 , . . .
Revised: In retrieval, the automatic system is required to return an ordered list of documents d 1 , d 2 , . . .

Original: , d n from a retrieval corpus R , given a query q .
Revised: , d n from a retrieval corpus R , given a query q .

Original: Both queries and documents are texts, i.e. , lists of tokens ⟨ t 1 , t 2 , . .
Revised: Both queries and documents are texts, i.e. , lists of tokens ⟨ t 1 , t 2 , . .

Original: , t N ⟩ , where each token t is drawn from a vocabulary V .
Revised: , t N ⟩ , where each token t is drawn from a vocabulary V .

Original: A span of tokens in a text is called an ngram; ngrams of size 1 are known as unigrams.
Revised: A span of tokens in a text is called an ngram; ngrams of size 1 are known as unigrams.

Original: We denote with F ( n, R ) the frequency of an ngram n in R , i.e. , the total number of times it appears in the whole retrieval corpus.
Revised: We denote with F ( n, R ) the frequency of an ngram n in R , i.e. , the total number of times it appears in the whole retrieval corpus.

Original: 3.1 The FM-Index
Revised: 3.1 The FM-Index

Original: Our method requires a data structure that can support the efficient identification of occurring substrings to guarantee that all decoded sequences are located somewhere in the retrieval corpus.
Revised: Our method requires a data structure that can support the efficient identification of occurring substrings to guarantee that all decoded sequences are located somewhere in the retrieval corpus.

Original: Moreover, to perform retrieval, we require the ability to identify which documents the generated ngrams appear in.
Revised: Moreover, to perform retrieval, we require the ability to identify which documents the generated ngrams appear in.

Original: Neither inverted indices (which have no efficient way to search for phrases of arbitrary length), nor prefix trees (which would force us to explicitly encode all k suffixes in a document), are viable options.
Revised: Neither inverted indices (which have no efficient way to search for phrases of arbitrary length), nor prefix trees (which would force us to explicitly encode all k suffixes in a document), are viable options.

Original: The core data structure that satisfies our requirements is the FM-index [Ferragina and Manzini, 2000], i.e. , a compressed suffix array that, as a self-index, requires no additional storage for the original text.
Revised: The core data structure that satisfies our requirements is the FM-index [Ferragina and Manzini, 2000], i.e. , a compressed suffix array that, as a self-index, requires no additional storage for the original text.

Original: FM-index space requirements are linear in the size of the corpus, and, with small vocabularies such as those used by modern subword-based language models, is thus usually significantly smaller than the uncompressed corpus.
Revised: FM-index space requirements are linear in the size of the corpus, and, with small vocabularies such as those used by modern subword-based language models, is thus usually significantly smaller than the uncompressed corpus.

Original: The FM-index can be used to count the frequency of any sequence of tokens n in O ( | n | log | V | ) , i.e. , independently from the size of the corpus itself.
Revised: The FM-index can be used to count the frequency of any sequence of tokens n in O ( | n | log | V | ) , i.e. , independently from the size of the corpus itself.

Original: For constrained decoding, the list of possible token successors can be obtained in O ( | V | log | V | ) .
Revised: For constrained decoding, the list of possible token successors can be obtained in O ( | V | log | V | ) .

Original: Internally, the FM-index relies on the Burrows-Wheeler Transform [Burrows and Wheeler, 1994], or BWT , an invertible transformation that permutes a string to make it easier to compress, defined as follows: all the rotations of the string are sorted lexicographically and laid out in a matrix; the last column of the matrix is the strings’s
Revised: Internally, the FM-index relies on the Burrows-Wheeler Transform [Burrows and Wheeler, 1994], or BWT , an invertible transformation that permutes a string to make it easier to compress, defined as follows: all the rotations of the string are sorted lexicographically and laid out in a matrix; the last column of the matrix is the strings’s

Original: BWT.
Revised: BWT.

Original: For example, given the string CABAC , the corresponding matrix would be:
Revised: For example, given the string CABAC , the corresponding matrix would be:

Original: F
Revised: F

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: where $ is a special end-of-string token.
Revised: where $ is a special end-of-string token.

Original: The first ( F ) and last ( L ) columns are the only ones that will be explicitly stored in the FM-index; F is just an array of runs ( i.e. , sequences of repeated tokens), due to the rotations being sorted, so it can be represented with one count for each alphabet symbol;
Revised: The first ( F ) and last ( L ) columns are the only ones that will be explicitly stored in the FM-index; F is just an array of runs ( i.e. , sequences of repeated tokens), due to the rotations being sorted, so it can be represented with one count for each alphabet symbol;

Original: L , the string’s BWT, will be stored in a data structure known as the Wavelet Tree [Grossi et al., 2003], which allows efficient rank-select-access queries, while exploiting the compressibility induced by the transformation.
Revised: L , the string’s BWT, will be stored in a data structure known as the Wavelet Tree [Grossi et al., 2003], which allows efficient rank-select-access queries, while exploiting the compressibility induced by the transformation.

Original: FM-indices have the useful property that for each symbol, the relative rank stays the same: that is, the i th occurrence of a symbol σ in F points to the same location in the corpus of the i th occurrence of σ in L .
Revised: FM-indices have the useful property that for each symbol, the relative rank stays the same: that is, the i th occurrence of a symbol σ in F points to the same location in the corpus of the i th occurrence of σ in L .

Original: Thanks to this property, we can locate any string ⟨ σ 1 , σ 2 , . . .
Revised: Thanks to this property, we can locate any string ⟨ σ 1 , σ 2 , . . .

Original: , σ n ⟩ in the index by starting from σ n and going backwards.
Revised: , σ n ⟩ in the index by starting from σ n and going backwards.

Original: First, we select the contiguous range of rows corresponding to the symbol σ n in F , then we check the ranks of the first and last occurrences of σ n − 1 in the same range of rows in L .
Revised: First, we select the contiguous range of rows corresponding to the symbol σ n in F , then we check the ranks of the first and last occurrences of σ n − 1 in the same range of rows in L .

Original: We use the ranks to select a new, smaller or equal range of rows looking up the symbol σ n − 1 in F .
Revised: We use the ranks to select a new, smaller or equal range of rows looking up the symbol σ n − 1 in F .

Original: The procedure can be applied iteratively to find ngrams of any size.
Revised: The procedure can be applied iteratively to find ngrams of any size.

Original:  Method
Revised:  Method

Original: In our retrieval methodology, SEAL, we generate multiple ngrams, conditioning on a query.
Revised: In our retrieval methodology, SEAL, we generate multiple ngrams, conditioning on a query.

Original: The ngrams are then used to find the documents they appear in within the corpus, which are then returned to the user.
Revised: The ngrams are then used to find the documents they appear in within the corpus, which are then returned to the user.

Original: In Figure 1 we show this process at a high-level.
Revised: In Figure 1 we show this process at a high-level.

Original: We use our indexing structure, i.e. , the
Revised: We use our indexing structure, i.e. , the

Original: FM-index to do constrained decoding so that each ngram occurs at least once in the retrieval corpus.
Revised: FM-index to do constrained decoding so that each ngram occurs at least once in the retrieval corpus.

Original: Jointly, we use the FM-index to efficiently find matching documents.
Revised: Jointly, we use the FM-index to efficiently find matching documents.

Original: Documents are ranked using the scores of the generated ngrams.
Revised: Documents are ranked using the scores of the generated ngrams.

Original: Autoregressive Retrieval We generate ngrams identifiers with constrained beam search, using the
Revised: Autoregressive Retrieval We generate ngrams identifiers with constrained beam search, using the

Original: FM-index to identify the set of possible next tokens in at most O ( | V | log | V | ) : tokens corresponding to unattested continuations are blocked by masking the logit to −∞ .
Revised: FM-index to identify the set of possible next tokens in at most O ( | V | log | V | ) : tokens corresponding to unattested continuations are blocked by masking the logit to −∞ .

Original: As a result, after a single decoding pass, we get a set of ngrams ( K ), along with their autoregressively-computed probabilities according to the model.
Revised: As a result, after a single decoding pass, we get a set of ngrams ( K ), along with their autoregressively-computed probabilities according to the model.

Original: It is also trivial to find the positions in the corpus where the decoded ngrams appear, as constrained decoding already requires selecting the relevant range of rows in the FM-index.
Revised: It is also trivial to find the positions in the corpus where the decoded ngrams appear, as constrained decoding already requires selecting the relevant range of rows in the FM-index.

Original: Note that autoregressive scoring entails monotonically decreasing scores—any string will be assigned a lower probability than any of its prefixes.
Revised: Note that autoregressive scoring entails monotonically decreasing scores—any string will be assigned a lower probability than any of its prefixes.

Original: To address this issue, we use fixed-length ngrams.
Revised: To address this issue, we use fixed-length ngrams.

Original: Each document is assigned the score ( P ( n | q ) ) of its most probable decoded occurring ngram, i.e. , the probability of the ngram as autoregressively computed by the decoder, conditioned on the encoder input We refer to this as the LM scoring.
Revised: Each document is assigned the score ( P ( n | q ) ) of its most probable decoded occurring ngram, i.e. , the probability of the ngram as autoregressively computed by the decoder, conditioned on the encoder input We refer to this as the LM scoring.

Original: Factoring in FM-index frequencies To counterbalance the monotonic probability decrease, we integrate in scoring unconditional ngram probabilities, computed as normalized index frequencies:
Revised: Factoring in FM-index frequencies To counterbalance the monotonic probability decrease, we integrate in scoring unconditional ngram probabilities, computed as normalized index frequencies:

Original: [Equation]
Revised: [Equation]

Original: This also enables us to promote distinctive ngrams, i.e. , those that have high probability according to the model and low probability according to the FM-index.
Revised: This also enables us to promote distinctive ngrams, i.e. , those that have high probability according to the model and low probability according to the FM-index.

Original: We take inspiration from the theory behind TF-IDF and BM25 [Robertson and Zaragoza, 2009] and use the following scoring function:
Revised: We take inspiration from the theory behind TF-IDF and BM25 [Robertson and Zaragoza, 2009] and use the following scoring function:

Original: [Equation]
Revised: [Equation]

Original: This formulation addresses the problem of length, as the unconditional probability of an ngram will also be equal or lower than that of any of its prefixes.
Revised: This formulation addresses the problem of length, as the unconditional probability of an ngram will also be equal or lower than that of any of its prefixes.

Original: To make better use of the computational resources, we slightly modify the beam search implementation to keep track of all the partially decoded sequences that have been considered.
Revised: To make better use of the computational resources, we slightly modify the beam search implementation to keep track of all the partially decoded sequences that have been considered.

Original: Thanks to this, we score a larger number of ngrams than the size of the beam.
Revised: Thanks to this, we score a larger number of ngrams than the size of the beam.

Original: We refer to this formulation as the LM+FM scoring.
Revised: We refer to this formulation as the LM+FM scoring.

Original: An Intersective Scoring for Multiple Ngrams One problem with the previous scoring formulations is that it is impossible to break ties among documents whose highest scoring ngram is the same, as they receive exactly the same score.
Revised: An Intersective Scoring for Multiple Ngrams One problem with the previous scoring formulations is that it is impossible to break ties among documents whose highest scoring ngram is the same, as they receive exactly the same score.

Original: Moreover, it might be difficult to capture all relevant information within a document by considering only a single ngram, for instance when salient ngrams are noncontiguous ( e.g. , separated by unrelated text).
Revised: Moreover, it might be difficult to capture all relevant information within a document by considering only a single ngram, for instance when salient ngrams are noncontiguous ( e.g. , separated by unrelated text).

Original: To address these issues we propose a novel scoring formulation that aggregates the contribution of multiple ngrams contained in the same document.
Revised: To address these issues we propose a novel scoring formulation that aggregates the contribution of multiple ngrams contained in the same document.

Original: To avoid repeated scoring of overlapping ngrams, for each document d ∈ R we only consider a subset of the generated ngrams K ( d ) ⊂ K .
Revised: To avoid repeated scoring of overlapping ngrams, for each document d ∈ R we only consider a subset of the generated ngrams K ( d ) ⊂ K .

Original: An ngram n belongs to K ( d ) if there is at least one occurrence of n in d that does not overlap in the corpus with an occurrence of another ngram n ′ such that a) n
Revised: An ngram n belongs to K ( d ) if there is at least one occurrence of n in d that does not overlap in the corpus with an occurrence of another ngram n ′ such that a) n

Original: ′ ∈ K
Revised: ′ ∈ K

Original: ( d ) b) w ( n ′ , q ) > w ( n, q ) .
Revised: ( d ) b) w ( n ′ , q ) > w ( n, q ) .

Original: The document-level score, then, is the weighted sum of all ngrams in K ( d ) :
Revised: The document-level score, then, is the weighted sum of all ngrams in K ( d ) :

Original: [Equation]
Revised: [Equation]

Original: where α is a hyperparameter and the coverage weight cover ( n, K ) (controlled by the second hyperparameter β ) is a function of how many ngram tokens are not included in the coverage set C ( n, K ) ⊂ V , i.e. , the union of all tokens in ngrams with a higher score.
Revised: where α is a hyperparameter and the coverage weight cover ( n, K ) (controlled by the second hyperparameter β ) is a function of how many ngram tokens are not included in the coverage set C ( n, K ) ⊂ V , i.e. , the union of all tokens in ngrams with a higher score.

Original: We define this coverage weight as follows:
Revised: We define this coverage weight as follows:

Original: [Equation]
Revised: [Equation]

Original: where set ( n ) is the set of all tokens in n .
Revised: where set ( n ) is the set of all tokens in n .

Original: The purpose of the coverage weight is to avoid the overscoring of very repetitive documents, where many similar ngrams are matched.
Revised: The purpose of the coverage weight is to avoid the overscoring of very repetitive documents, where many similar ngrams are matched.

Original: Note that by saving the probability distribution at the first decoding step we can compute scores for all unigrams with no additional forward pass.
Revised: Note that by saving the probability distribution at the first decoding step we can compute scores for all unigrams with no additional forward pass.

Original: We refer to this last approach, which can be thought of as a higher-order generalization of the bag-of-words assumption, as the LM+FM intersective scoring.
Revised: We refer to this last approach, which can be thought of as a higher-order generalization of the bag-of-words assumption, as the LM+FM intersective scoring.

Original:  Experimental Setting
Revised:  Experimental Setting

Original: Our experimental setting evaluates SEAL on English knowledge-intensive NLP tasks.
Revised: Our experimental setting evaluates SEAL on English knowledge-intensive NLP tasks.

Original: Each considered dataset is a collection of queries, each of which can be answered by looking for piece(s) of evidence in the corpus.
Revised: Each considered dataset is a collection of queries, each of which can be answered by looking for piece(s) of evidence in the corpus.

Original: We consider both an in vivo evaluation, in which we assess the model by looking at how well the document ranking matches with the ground truth, and, in addition, we perform a downstream evaluation, in which we feed the retrieved documents to a trained reader, that uses the documents to generate the answer.
Revised: We consider both an in vivo evaluation, in which we assess the model by looking at how well the document ranking matches with the ground truth, and, in addition, we perform a downstream evaluation, in which we feed the retrieved documents to a trained reader, that uses the documents to generate the answer.

Original: 5.1 Data
Revised: 5.1 Data

Original: Natural Questions Natural Questions (NQ) is dataset containing query-document pairs, where the query is a question ( e.g. , “who wrote photograph by ringo starr”), and the document is a Wikipedia page, in which a span is marked as an answer [Kwiatkowski et al., 2019].
Revised: Natural Questions Natural Questions (NQ) is dataset containing query-document pairs, where the query is a question ( e.g. , “who wrote photograph by ringo starr”), and the document is a Wikipedia page, in which a span is marked as an answer [Kwiatkowski et al., 2019].

Original: We experiment on both the customary retrieval setup used by, among others, Karpukhin et al.
Revised: We experiment on both the customary retrieval setup used by, among others, Karpukhin et al.

Original: [2020] and Mao et al.
Revised: [2020] and Mao et al.

Original: [2021], and the substantially different setup used by Tay et al.
Revised: [2021], and the substantially different setup used by Tay et al.

Original: We refer to these two settings as, respectively, NQ and NQ320 k .
Revised: We refer to these two settings as, respectively, NQ and NQ320 k .

Original: In NQ, retrieval is performed on an entire Wikipedia dump, chunked in around 21 M passages of 100 tokens.
Revised: In NQ, retrieval is performed on an entire Wikipedia dump, chunked in around 21 M passages of 100 tokens.

Original: Performance is measured as accuracy@ k , i.e. , the fraction of instances for which at least one of the top- k retrieved passages contains the answer.
Revised: Performance is measured as accuracy@ k , i.e. , the fraction of instances for which at least one of the top- k retrieved passages contains the answer.

Original: NQ320 k is a much more restricted setting, in which the retrieval set is limited to the union of all ground truth document in the training, dev or test set.
Revised: NQ320 k is a much more restricted setting, in which the retrieval set is limited to the union of all ground truth document in the training, dev or test set.

Original: Different revisions of the same Wikipedia page count as different documents.
Revised: Different revisions of the same Wikipedia page count as different documents.

Original: Note that the exact splits used by Tay et al.
Revised: Note that the exact splits used by Tay et al.

Original: [2022], the retrieval corpus and the preprocessing code have not been yet released at the time of writing.
Revised: [2022], the retrieval corpus and the preprocessing code have not been yet released at the time of writing.

Original: Therefore, we have tried to replicate the setting as closely as possible, but the exact numbers are not precisely comparable with those reported in the original paper.
Revised: Therefore, we have tried to replicate the setting as closely as possible, but the exact numbers are not precisely comparable with those reported in the original paper.

Original: In NQ320 k , performance is measured as hits@ k , i.e, the fraction of instances for which at least one of the top- k retrieved passages is in the ground truth.
Revised: In NQ320 k , performance is measured as hits@ k , i.e, the fraction of instances for which at least one of the top- k retrieved passages is in the ground truth.

Original: KILT is a comprehensive benchmark collecting different datasets including question answering, fact checking, dialogue, slot filling, and entity linking [Petroni et al., 2021].
Revised: KILT is a comprehensive benchmark collecting different datasets including question answering, fact checking, dialogue, slot filling, and entity linking [Petroni et al., 2021].

Original: All these tasks are solvable by retrieving information from a unified corpus — a Wikipedia dump.
Revised: All these tasks are solvable by retrieving information from a unified corpus — a Wikipedia dump.

Original: In KILT, the evidence is usually the paragraph that contains the answer.
Revised: In KILT, the evidence is usually the paragraph that contains the answer.

Original: Following Maillard et al.
Revised: Following Maillard et al.

Original: [2021], we have rechunked KILT’s retrieval corpus, which is originally paragraph-based, in around 36 M passages of 100 tokens.
Revised: [2021], we have rechunked KILT’s retrieval corpus, which is originally paragraph-based, in around 36 M passages of 100 tokens.

Original: We do not use the entity linking and ELI5 KILT tasks, where a ground truth passage is not provided in the training set.
Revised: We do not use the entity linking and ELI5 KILT tasks, where a ground truth passage is not provided in the training set.

Original: KILT’s retrieval performance is measured with R-precision, a precision-oriented measure that considers only gold documents as correct answers, not just any document containing the answer.
Revised: KILT’s retrieval performance is measured with R-precision, a precision-oriented measure that considers only gold documents as correct answers, not just any document containing the answer.

Original: R-precision can be computed at either passage level or at page level.
Revised: R-precision can be computed at either passage level or at page level.

Original: [Table]
Revised: [Table]

Original: 5. SEAL configuration
Revised: 5. SEAL configuration

Original: Training We finetune BART large [Lewis et al., 2019] to generate ngrams of length k = 10 from the ground truth document.
Revised: Training We finetune BART large [Lewis et al., 2019] to generate ngrams of length k = 10 from the ground truth document.

Original: Since there are | d | − k ngrams in a document d , we sample (with replacement) 10 ngrams from it, biasing the distribution in favor of ngrams with a high character overlap with the query.
Revised: Since there are | d | − k ngrams in a document d , we sample (with replacement) 10 ngrams from it, biasing the distribution in favor of ngrams with a high character overlap with the query.

Original: We also add the title of the document to the set of training ngrams.
Revised: We also add the title of the document to the set of training ngrams.

Original: To expose the model to more possible pieces of evidence, we also add different “unsupervised” examples for each document in the retrieval corpus to the training set.
Revised: To expose the model to more possible pieces of evidence, we also add different “unsupervised” examples for each document in the retrieval corpus to the training set.

Original: In each of these examples the model takes as input a uniformly sampled span from the document, and predicts either another sampled span, or the title of the page.
Revised: In each of these examples the model takes as input a uniformly sampled span from the document, and predicts either another sampled span, or the title of the page.

Original: We append special tokens to the input to signal to the model a) whether the pair comes from the supervised or unsupervised training pairs (in the same spirit as the co-training task prompts used by Tay et al.
Revised: We append special tokens to the input to signal to the model a) whether the pair comes from the supervised or unsupervised training pairs (in the same spirit as the co-training task prompts used by Tay et al.

Original: [2022]) b) whether a title or span is expected as output.
Revised: [2022]) b) whether a title or span is expected as output.

Original: On KILT we train
Revised: On KILT we train

Original: SEAL on all datasets at once.
Revised: SEAL on all datasets at once.

Original: We report training and inference hyperameters in Appendix (§A).
Revised: We report training and inference hyperameters in Appendix (§A).

Original: How to choose ngrams In preliminary experiments, we have found that training SEAL with ngrams of fixed length sampled uniformly from the training chunk did not lead to good performances.
Revised: How to choose ngrams In preliminary experiments, we have found that training SEAL with ngrams of fixed length sampled uniformly from the training chunk did not lead to good performances.

Original: Instead, we have biased the ngram distribution towards ngrams that were relevant to the query.
Revised: Instead, we have biased the ngram distribution towards ngrams that were relevant to the query.

Original: As a simple proxy for relevance, we have use the Levenshtein (character-based) distance between the query and the ngram overlap.
Revised: As a simple proxy for relevance, we have use the Levenshtein (character-based) distance between the query and the ngram overlap.

Original: The final distribution to sample from is defined by the following formula:
Revised: The final distribution to sample from is defined by the following formula:

Original: [Equation]
Revised: [Equation]

Original: where L is the Levenshtein distance and τ ( = 1 . 5 in our experiments) is a temperature parameter, controlling the peakiness of the distribution.
Revised: where L is the Levenshtein distance and τ ( = 1 . 5 in our experiments) is a temperature parameter, controlling the peakiness of the distribution.

Original: Index We use the C++ FM-index implementation in sdsl-lite .
Revised: Index We use the C++ FM-index implementation in sdsl-lite .

Original: While the FM-index construction (which requires a sort of all rotations) takes around 6 hours in our single-threaded implementation, parallel algorithms are available [Labeit et al., 2017].
Revised: While the FM-index construction (which requires a sort of all rotations) takes around 6 hours in our single-threaded implementation, parallel algorithms are available [Labeit et al., 2017].

Original: Each document is encoded as the subword tokenization of the concatenation of the title and the passage, separated by a special token.
Revised: Each document is encoded as the subword tokenization of the concatenation of the title and the passage, separated by a special token.

Original: We report in Table 1 the index statistics for Natural Questions.
Revised: We report in Table 1 the index statistics for Natural Questions.

Original: As can be seen, SEAL’s FM-index is more than 7 times lighter compared to DPR’s full document embeddings for exact inner product search, and needs neither a GPU for search on top of that, nor separate storage for the text itself.
Revised: As can be seen, SEAL’s FM-index is more than 7 times lighter compared to DPR’s full document embeddings for exact inner product search, and needs neither a GPU for search on top of that, nor separate storage for the text itself.

Original: While vector compression methods can reduce dense retrievers’ index size, this still comes at the expense of performance [Yamada et al., 2021, Lewis et al., 2021a].
Revised: While vector compression methods can reduce dense retrievers’ index size, this still comes at the expense of performance [Yamada et al., 2021, Lewis et al., 2021a].

Original: In addition, our the size of our index is less than 50% of that of the well-optimized Lucene BM25 index used by pyserini , but also roughly 65% of the uncompressed plain text itself.
Revised: In addition, our the size of our index is less than 50% of that of the well-optimized Lucene BM25 index used by pyserini , but also roughly 65% of the uncompressed plain text itself.

Original: 5.3 Retriever Baselines
Revised: 5.3 Retriever Baselines

Original: We compare SEAL against well-established systems in the literature on each benchmark.
Revised: We compare SEAL against well-established systems in the literature on each benchmark.

Original: On NQ and
Revised: On NQ and

Original: NQ320 k
Revised: NQ320 k

Original: we also compare against our BART-based replication of DSI [Tay et al., 2022, DSI-BART ].
Revised: we also compare against our BART-based replication of DSI [Tay et al., 2022, DSI-BART ].

Original: On NQ320k, a page-level benchmark, we include our own replication of GENRE [De Cao et al., 2021b].
Revised: On NQ320k, a page-level benchmark, we include our own replication of GENRE [De Cao et al., 2021b].

Original: Unless otherwise specified, we use pyserini to compute the BM25 baseline.
Revised: Unless otherwise specified, we use pyserini to compute the BM25 baseline.

Original: For other systems, we either take figures from the literature, or use publicly released model predictions.
Revised: For other systems, we either take figures from the literature, or use publicly released model predictions.

Original: [Table]
Revised: [Table]

Original: [Table] 5.4 Reader
Revised: [Table] 5.4 Reader

Original: For downstream results, we use the Fusion-in-Decoder abstractive reader [Izacard and Grave, 2021], which takes in the query along with 100 contexts and produces a task-specific answer.
Revised: For downstream results, we use the Fusion-in-Decoder abstractive reader [Izacard and Grave, 2021], which takes in the query along with 100 contexts and produces a task-specific answer.

Original: We train FiD on training set predictions.
Revised: We train FiD on training set predictions.

Original:  Results NQ320 k
Revised:  Results NQ320 k

Original: We report results on NQ320 k in Table 2.
Revised: We report results on NQ320 k in Table 2.

Original: SEAL outperforms BM25 and DSI-BART in hits@10 in all its formulations.
Revised: SEAL outperforms BM25 and DSI-BART in hits@10 in all its formulations.

Original: When taking into account ngram frequencies ( i.e. , LM+FM),
Revised: When taking into account ngram frequencies ( i.e. , LM+FM),

Original: SEAL achieves even higher results than GENRE, despite the fact that this benchmark only requires page-level retrieval capabilities (that is the focus of GENRE).
Revised: SEAL achieves even higher results than GENRE, despite the fact that this benchmark only requires page-level retrieval capabilities (that is the focus of GENRE).

Original: Finally, our intersective formulation achieves the highest results, both in hits@1 and @10, indicating that multiple ngrams identifiers might capture complementary information, which can be aggregated for stronger performances.
Revised: Finally, our intersective formulation achieves the highest results, both in hits@1 and @10, indicating that multiple ngrams identifiers might capture complementary information, which can be aggregated for stronger performances.

Original: Natural Questions We report in Table 

Revised: Natural Questions We report in Table

Original: the results of our evaluation on Natural Questions, a passage-level retrieval benchmark with a larger collection of documents ( i.e. , ~ 21 M w.r.t.
Revised: the results of our evaluation on Natural Questions, a passage-level retrieval benchmark with a larger collection of documents ( i.e. , ~ 21 M w.r.t.

Original: 200 k in
Revised: 200 k in

Original: NQ320 k ).
Revised: NQ320 k ).

Original: In this setting, the gap in performance between DSI-BART and SEAL is larger, possibly because memorizing documents identifiers in the parameters of the model becomes more challenging with larger corpora.
Revised: In this setting, the gap in performance between DSI-BART and SEAL is larger, possibly because memorizing documents identifiers in the parameters of the model becomes more challenging with larger corpora.

Original: Remarkably, the intersective formulation of SEAL achieves results comparable or superior to more established retrieval paradigms ( e.g. , BM25, DPR and GAR), at high-recall (accuracy@100).
Revised: Remarkably, the intersective formulation of SEAL achieves results comparable or superior to more established retrieval paradigms ( e.g. , BM25, DPR and GAR), at high-recall (accuracy@100).

Original: To better understand the generalization capabilities of our retrieval solution we use the question/answer overlap split of Lewis et al.
Revised: To better understand the generalization capabilities of our retrieval solution we use the question/answer overlap split of Lewis et al.

Original: [2021b].
Revised: [2021b].

Original: This study reveals that SEAL achieves the highest performance for question/answer pairs never seen during training ( i.e. , no overlap), suggesting a better ability to generalize to completely novel questions with novel answers ( e.g. , 3 . 5 points better than GAR on average).
Revised: This study reveals that SEAL achieves the highest performance for question/answer pairs never seen during training ( i.e. , no overlap), suggesting a better ability to generalize to completely novel questions with novel answers ( e.g. , 3 . 5 points better than GAR on average).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: KILT We report retrieval results at passage level on the KILT benchmark in Table 4.
Revised: KILT We report retrieval results at passage level on the KILT benchmark in Table 4.

Original: 3 SEAL outperforms DPR by more than 10 points on average in passage-level R-precision, indicating that our method is more precise in surfacing ground truth evidence as the first result.
Revised: 3 SEAL outperforms DPR by more than 10 points on average in passage-level R-precision, indicating that our method is more precise in surfacing ground truth evidence as the first result.

Original: Moreover, SEAL also performs better than MT-DPR (multi-task DPR) even when the latter is pretrained on tens of millions of questions from PAQ [Lewis et al., 2021c], a technique that can drastically improve results and that could potentially bring benefits to our method as well (a task we leave for future work).
Revised: Moreover, SEAL also performs better than MT-DPR (multi-task DPR) even when the latter is pretrained on tens of millions of questions from PAQ [Lewis et al., 2021c], a technique that can drastically improve results and that could potentially bring benefits to our method as well (a task we leave for future work).

Original: When it comes to downstream performances (Table 5), FiD with passages retrieved by intersective SEAL establishes a new state-of-the-art on 4 datasets out of 7 (FEVER, zsRE, NQ, HoPo), and achieves very competitive results on the remaining 3.
Revised: When it comes to downstream performances (Table 5), FiD with passages retrieved by intersective SEAL establishes a new state-of-the-art on 4 datasets out of 7 (FEVER, zsRE, NQ, HoPo), and achieves very competitive results on the remaining 3.

Original: Speed and constrained decoding The inference speed of SEAL is directly proportional to the beam size, with a limited overhead added by constrained decoding.
Revised: Speed and constrained decoding The inference speed of SEAL is directly proportional to the beam size, with a limited overhead added by constrained decoding.

Original: On the Natural Questions test set, for instance, retrieval with the intersective scoring requires on our 1 GPU evaluation setup ~16 minutes and ~35 minutes with, respectively, a beam size of 5 or 15 .
Revised: On the Natural Questions test set, for instance, retrieval with the intersective scoring requires on our 1 GPU evaluation setup ~16 minutes and ~35 minutes with, respectively, a beam size of 5 or 15 .

Original: Mao et al.
Revised: Mao et al.

Original: [2021] report a lower runtime for GAR (~5 minutes), and a comparable one for DPR (~30 minutes).
Revised: [2021] report a lower runtime for GAR (~5 minutes), and a comparable one for DPR (~30 minutes).

Original: Note that more efficient approaches to constrained decoding have been proposed ( e.g. , De Cao et al.
Revised: Note that more efficient approaches to constrained decoding have been proposed ( e.g. , De Cao et al.

Original: [2021a])
Revised: [2021a])

Original: [Table]
Revised: [Table]

Original: [Table] and we leave their application to SEAL as future work.
Revised: [Table] and we leave their application to SEAL as future work.

Original: Moreover, generation is becoming the de facto standard approach to NLP, not just as the method for materializing final outputs, but also for modeling the computational process needed before computing the answer [Wei et al., 2022].
Revised: Moreover, generation is becoming the de facto standard approach to NLP, not just as the method for materializing final outputs, but also for modeling the computational process needed before computing the answer [Wei et al., 2022].

Original: As such, we expect generation latency will improve significantly and increasingly over time as a result of this growing interest.
Revised: As such, we expect generation latency will improve significantly and increasingly over time as a result of this growing interest.

Original: Any improvement will be directly applicable to SEAL
Revised: Any improvement will be directly applicable to SEAL

Original: [Table]
Revised: [Table]

Original: Model size In Table 4 we show that parameter size is not a crucial factor behind the good performance of the method that we propose: SEAL (~400M) outperforms all model from O˘guz et al.
Revised: Model size In Table 4 we show that parameter size is not a crucial factor behind the good performance of the method that we propose: SEAL (~400M) outperforms all model from O˘guz et al.

Original: [2021], including the MT-DPR large model (~350M) trained on NQ and PAQ [Lewis et al., 2021c].
Revised: [2021], including the MT-DPR large model (~350M) trained on NQ and PAQ [Lewis et al., 2021c].

Original: In fact, the results of MT-DPR are lower with a bigger backbone—which could point to an increased difficulty in training larger models for dense retrieval.
Revised: In fact, the results of MT-DPR are lower with a bigger backbone—which could point to an increased difficulty in training larger models for dense retrieval.

Original: In the same direction, on vanilla NQ (Table 6), where DPR large only slightly outperforms DPR base, the performance of SEAL is in the same ballpark.
Revised: In the same direction, on vanilla NQ (Table 6), where DPR large only slightly outperforms DPR base, the performance of SEAL is in the same ballpark.

Original: Ablation studies In Table 7 we report results on NQ for various configurations of SEAL.
Revised: Ablation studies In Table 7 we report results on NQ for various configurations of SEAL.

Original: While, in general, performances increase with a larger beam, diminishing returns (or even a performance decrease) are found between a value of 10 and 15 .
Revised: While, in general, performances increase with a larger beam, diminishing returns (or even a performance decrease) are found between a value of 10 and 15 .

Original: Disabling constrained decoding and discarding a posteriori all generated ngrams that don’t appear in the corpus results in slightly lower performances.
Revised: Disabling constrained decoding and discarding a posteriori all generated ngrams that don’t appear in the corpus results in slightly lower performances.

Original: We have found the decoding maximum length to have a crucial impact on the retrieval performance of SEAL, since shorter ngrams tend to be less informative than longer ones.
Revised: We have found the decoding maximum length to have a crucial impact on the retrieval performance of SEAL, since shorter ngrams tend to be less informative than longer ones.

Original: We report in Tableperformances on the NQ test set when using, respectivaly, 3, 5, and 10 as ngram maximum length.
Revised: We report in Tableperformances on the NQ test set when using, respectivaly, 3, 5, and 10 as ngram maximum length.

Original: Every other SEAL figure reported in this paper uses 10.
Revised: Every other SEAL figure reported in this paper uses 10.

Original: Qualitative Analysis In Table 9, we show examples of ngrams predicted by SEAL (trained on
Revised: Qualitative Analysis In Table 9, we show examples of ngrams predicted by SEAL (trained on

Original: KILT) given the query “can you predict earthquakes”.
Revised: KILT) given the query “can you predict earthquakes”.

Original: SEAL is able to rephrase the query in ways that preserve its lexical material producing ngrams such as earthquakes can be predicted , used to predict earthquakes etc.
Revised: SEAL is able to rephrase the query in ways that preserve its lexical material producing ngrams such as earthquakes can be predicted , used to predict earthquakes etc.

Original: Morevoer, the model is also able to explore more diverse regions of the output space, overcoming the vocabulary mismatch problem: ngrams contain related tokens like the subword seism-
Revised: Morevoer, the model is also able to explore more diverse regions of the output space, overcoming the vocabulary mismatch problem: ngrams contain related tokens like the subword seism-

Original: and the word forecast .
Revised: and the word forecast .

Original: SEAL’s LM+FM scoring is also able to assign a score below 0 (and, thus, exclude from the search), unrelated ngrams that are considered by the beam because of their promising start, such as “Seismic risk in Malta @@”.
Revised: SEAL’s LM+FM scoring is also able to assign a score below 0 (and, thus, exclude from the search), unrelated ngrams that are considered by the beam because of their promising start, such as “Seismic risk in Malta @@”.

Original: [Table]
Revised: [Table]

Original:  Discussion
Revised:  Discussion

Original: With SEAL we present solution that could potentially find applications outside information retrieval ( e.g. , enforce generated substrings come from a white list of trusted sources).
Revised: With SEAL we present solution that could potentially find applications outside information retrieval ( e.g. , enforce generated substrings come from a white list of trusted sources).

Original: While we conduct our experiments with a model of ~ 400 M parameters ( i.e. , BART) for fast iterations, we believe the use of larger models could considerably improve performance.
Revised: While we conduct our experiments with a model of ~ 400 M parameters ( i.e. , BART) for fast iterations, we believe the use of larger models could considerably improve performance.

Original: Changing the model would not affect the size of the index nor the cost of using it — O ( | n | log | V | ) for finding an ngram n .
Revised: Changing the model would not affect the size of the index nor the cost of using it — O ( | n | log | V | ) for finding an ngram n .

Original: Moreover, we believe that indexing very large corpora ( e.g. , the web) could be done more efficiently than existing attempts ( e.g. , Piktus et al.
Revised: Moreover, we believe that indexing very large corpora ( e.g. , the web) could be done more efficiently than existing attempts ( e.g. , Piktus et al.

Original: [2021]) given the light memory footprint.
Revised: [2021]) given the light memory footprint.

Original: Finally, dynamic variants [Gerlach, 2007,
Revised: Finally, dynamic variants [Gerlach, 2007,

Original: Salson et al., 2009] could allow the update of the FM-index on the fly without the need of re-indexing.
Revised: Salson et al., 2009] could allow the update of the FM-index on the fly without the need of re-indexing.

Original: While out of the scope of the current paper, we plan to tackle some of these scaling challenges in future work.
Revised: While out of the scope of the current paper, we plan to tackle some of these scaling challenges in future work.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we present SEAL, a novel retrieval system that combines an autoregressive language model with a compressed full-text substring index.
Revised: In this paper we present SEAL, a novel retrieval system that combines an autoregressive language model with a compressed full-text substring index.

Original: Such combination allows to constrain the generation to existing ngrams in a corpus and to jointly retrieve documents containing them.
Revised: Such combination allows to constrain the generation to existing ngrams in a corpus and to jointly retrieve documents containing them.

Original: Empirically, we show an improvement by more than 10 points in average passage-level R-precision on KILT, and establish new state-of-the-art downstream performance on 4 out 7 datasets when paired with a reader model.
Revised: Empirically, we show an improvement by more than 10 points in average passage-level R-precision on KILT, and establish new state-of-the-art downstream performance on 4 out 7 datasets when paired with a reader model.

Original: While our results show that SEAL could already compete with more established retrieval systems, we believe there is potential in exploring existing (or yet to come) larger autoregressive models.
Revised: While our results show that SEAL could already compete with more established retrieval systems, we believe there is potential in exploring existing (or yet to come) larger autoregressive models.

Original:  Broader Impact
Revised:  Broader Impact

Original: While our decoding methodology provides a way to enforce corpus-based generation constraints, this does not fully prevent the generation and retrieval of non-factual or abusive text, since the corpus itself may well contain non-factual or abusive text.
Revised: While our decoding methodology provides a way to enforce corpus-based generation constraints, this does not fully prevent the generation and retrieval of non-factual or abusive text, since the corpus itself may well contain non-factual or abusive text.

Original: Moreover, even if a given corpus is misinformative or abusive, the language model could still produce undesirable text by “misquoting” material from it, e.g. , selectively copying an utterance that is being criticized in the original context of appearance.
Revised: Moreover, even if a given corpus is misinformative or abusive, the language model could still produce undesirable text by “misquoting” material from it, e.g. , selectively copying an utterance that is being criticized in the original context of appearance.

Original: Applications should allow user to check the original supporting text.
Revised: Applications should allow user to check the original supporting text.

Original: Furthermore, while we have only used a relatively small language model, scaling up the autoregressive model of SEAL would come with the same environmental risks that any other large system would pose, due to the more demanding energy requirements.
Revised: Furthermore, while we have only used a relatively small language model, scaling up the autoregressive model of SEAL would come with the same environmental risks that any other large system would pose, due to the more demanding energy requirements.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank Aleksandra Piktus, Edoardo Barba, Niccolò Campolungo, and Pere-Lluis Huguet Cabot for their helpful comments and suggestions.
Revised: We thank Aleksandra Piktus, Edoardo Barba, Niccolò Campolungo, and Pere-Lluis Huguet Cabot for their helpful comments and suggestions.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Discussion in the Appendix (§9) (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] Discussion in the Appendix (§9) (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The prediction code is available in the supplementary materials.
Revised: [Yes] The prediction code is available in the supplementary materials.

Original: For training, we used the popular fairseq , library.
Revised: For training, we used the popular fairseq , library.

Original: We have reported hyperparameters in §5.2 and in the Appendix (§A).
Revised: We have reported hyperparameters in §5.2 and in the Appendix (§A).

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] 4.
Revised: [No] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [No] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] 5.
Revised: [No] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A][Table] [Table] Appendix A Hyperparameters
Revised: [N/A]

Original: Training We finetune the model using fairseq .

Original: We use Adam [Kingma and Ba, 2015] with a learning rate of 3 · 10 − 5 , warming up for 500 updates, then using polynomial decay for at 800 k updates, evaluating every 15 k steps.

Original: We stop the training run if the loss on the development set stops improving for 5 evaluation passes.

Original: We use label smoothing ( 0 . 1 ), weight decay ( 0 . 01 ), and gradient norm clipping ( 0 . 1 ).

Original: We train in batches of 4096 tokens on 8 GPUs.

Original: Inference We decode for 10 timesteps with a beam size of 15 , and set the hyperparameters α , and β to, respectively, 2 . 0 and 0 .

Original: The hyperparameters have been tuned on the Natural Questions development set.

Original: In the constrained decoding stage, we force part of the generated ngrams to match document titles.

Original: B DSI-BART replication details

Original: On NQ320 k , bert-base-cased is used to compute the embeddings for the clustering.

Original: On regular

Original: NQ, we use the public precomputed DPR embeddings.

Original: To compare fairly against SEAL, we fine-tune the same encoder-decoder backbone, i.e. , BART large.

Original: C Additional KILT results

Original: We report in Table 10 page-level results on the KILT test set.

Original: On most datasets, SEAL obtains results which are comparable or better than other systems performing page-level retrieval.

Original: Furthermore, results are within two points of the average performance of GENRE, i.e. , a system that directly targets the page-level setting.

Original: Comparing KILT-scores (Table 11), i.e. , a metric combining downstream performances and page-level R-precision, we achieve state-of-the-art results on 4 out of 7 datasets.

Original: [Table]

Original: when using

Original: [Table]

Original: [Table]

Original: not using (

Original: ) super- data

Original: Reporting accuracy@

Original: [Table]