Original: T OWARDS U NSUPERVISED D ENSE I NFORMATION R ETRIEVAL WITH C ONTRASTIVE L EARNING
Revised: T OWARDS U NSUPERVISED D ENSE I NFORMATION R ETRIEVAL WITH C ONTRASTIVE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking.
Revised: Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking.

Original: Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency.
Revised: Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency.

Original: These models have obtained state-of-the-art results on datasets and tasks where large training sets are available.
Revised: These models have obtained state-of-the-art results on datasets and tasks where large training sets are available.

Original: However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised.
Revised: However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised.

Original: Thus, a natural question is whether it is possible to train dense retrievers without supervision.
Revised: Thus, a natural question is whether it is possible to train dense retrievers without supervision.

Original: In this work, we consider contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance.
Revised: In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance.

Original: More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 8 out of 14 datasets.
Revised: More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 8 out ofdatasets.

Original: Furthermore, when a few thousands examples are available, we show that ﬁnetuning our model on these leads to strong improvements compared to BM25.
Revised: Furthermore, when a few thousands examples are available, we show that ﬁne-tuning our model on these leads to strong improvements compared to BM25.

Original: Finally, when used as pre-training before ﬁne-tuning on the MS MARCO dataset, our technique obtains state-of-the-art results on the BEIR benchmark.
Revised: Finally, when used as pre-training before ﬁne-tuning on the MS MARCO dataset, our technique obtains state-of-the-art results on the BEIR benchmark.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Document retrieval is the task of ﬁnding relevant documents in a large collection to answer speciﬁc queries.
Revised: Document retrieval is the task of ﬁnding relevant documents in a large collection to answer speciﬁc queries.

Original: This is an important task by itself and a core component of many natural language processing (NLP) problems, such as open domain question answering (Chen et al., 2017) or fact checking (Thorne et al., 2018).
Revised: This is an important task by itself and a core component of many natural language processing (NLP) problems, such as open domain question answering (Chen et al., 2017) or fact checking (Thorne et al., 2018).

Original: Traditionally, retrieval systems, or retrievers, leverage lexical similarities to match queries and documents, using, for instance, TF-IDF or BM25 weighting (Robertson & Zaragoza, 2009).
Revised: Traditionally, retrieval systems, or retrievers, leverage lexical similarities to match queries and documents, using, for instance, TF-IDF or BM25 weighting (Robertson & Zaragoza, 2009).

Original: These approaches, based on exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000).
Revised: These approaches, based on exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000).

Original: By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-ofthe-art performance on question answering benchmarks, such as MS-MARCO (Nguyen et al., 2016) or NaturalQuestion (Kwiatkowski et al., 2019).
Revised: By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-ofthe-art performance on question answering benchmarks, such as MS-MARCO (Nguyen et al., 2016) or NaturalQuestion (Kwiatkowski et al., 2019).

Original: As training neural networks requires large amount of data, their strong retrieval results were possible for domains and applications where large training datasets are available.
Revised: As training neural networks requires large amount of data, their strong retrieval results were possible for domains and applications where large training datasets are available.

Original: In the case of retrieval, creating these datasets requires manually matching queries to the relevant documents in the collection.
Revised: In the case of retrieval, creating these datasets requires manually matching queries to the relevant documents in the collection.

Original: This is hardly possible when the collection contains millions or billions of element, resulting in many scenarios where only a few in-domain examples, if any, are available.
Revised: This is hardly possible when the collection contains millions or billions of element, resulting in many scenarios where only a few in-domain examples, if any, are available.

Original: A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting refered to as zero-shot .
Revised: A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting refered to as zero-shot .

Original: Unfortunately, in this setting dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al., 2021).
Revised: Unfortunately, in this setting dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al., 2021).

Original: Unsupervised learning is one alternative to transfer learning that could alleviate the above problems– and also serve as strong basis for further ﬁne-tuning.
Revised: Thus, a natural alternative to transfer learning is unsupervised learning, which raises the following question: is it possible to train dense retrievers without supervision, and match the performance of

Original: In fact, signiﬁcant progress has been made in this direction, with training objective such as ICT (Lee et al., 2019) showing great promise. However, to date there is no evidence that it can match the performance of unsupervised adhoc retrieval methods like BM25, let alone beat it. The goal of this paper is to understand if and how this gap could be bridged.
Revised: BM25?

Original: Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval.
Revised: Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval.

Original: Given a document, one can generate a synthetic query and then train the network to retrieve the original document, among many others, given the query.
Revised: Given a document, one can generate a synthetic query and then train the network to retrieve the original document, among many others, given the query.

Original: The inverse Cloze task (ICT), proposed by Lee et al. (2019) to pre-train retrievers, uses a given sentence as query and predict the context surrounding it.
Revised: The inverse Cloze task (ICT), proposed by Lee et al. (2019) to pre-train retrievers, uses a given sentence as query and predict the context surrounding it.

Original: While showing promising results as pre-training (Chang et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when used as a zero-shot retriever.
Revised: While showing promising results as pre-training (Chang et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when used as a zero-shot retriever.

Original: ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020;
Revised: ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020;

Original: He et al., 2020).
Revised: He et al., 2020).

Original: In particular, computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval (Caron et al., 2021).
Revised: In particular, computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval (Caron et al., 2021).

Original: In this work, we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision.
Revised: In this work, we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision.

Original: We want to investigate how much the recent developments from computer vision, such as MoCo, can improve dense retrievers.
Revised: We want to investigate how much the recent developments from computer vision, such as MoCo, can improve dense retrievers.

Original: Here, our goal is not to develop new techniques, but to determine how far we can go by pushing existing methods to their limits.
Revised: Here, our goal is not to develop new techniques, but to determine how far we can go by pushing existing methods to their limits.

Original: In this paper, we thus make the following contributions.
Revised: In this paper, we thus make the following contributions.

Original: First, we show that contrastive learning can lead to strong unsupervised retrievers: our model achieves Recall@100 results competitive with
Revised: First, we show that contrastive learning can lead to strong unsupervised retrievers: our model achieves Recall@100 results competitive with

Original: BM25 on most of the BEIR benchmark.
Revised: BM25 on most of the BEIR benchmark.

Original: Second, in a few-shot setting, we show that our model beneﬁts from few training examples, and obtains better results than transfering models from large datasets such as MS MARCO.
Revised: Second, in a few-shot setting, we show that our model beneﬁts from few training examples, and obtains better results than transfering models from large datasets such as MS MARCO.

Original: Third, when used as a pre-training method before ﬁne-tuning on
Revised: Third, when used as a pre-training method before ﬁne-tuning on

Original: MS MARCO, our technique leads to state-of-the-art retrieval performance on the BEIR benchmark.
Revised: MS MARCO, our technique leads to state-of-the-art retrieval performance on the BEIR benchmark.

Original: Finally, we perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task.
Revised: Finally, we perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In this section, we brieﬂy review relevant work in information retrieval, and application of machine learning to this problem.
Revised: In this section, we brieﬂy review relevant work in information retrieval, and application of machine learning to this problem.

Original: This is not an exhaustive review, and we refer the reader to Manning et al.
Revised: This is not an exhaustive review, and we refer the reader to Manning et al.

Original: (2008), Mitra et al.
Revised: (2008), Mitra et al.

Original: (2018) and Lin et al.
Revised: (2018) and Lin et al.

Original: for a more complete introduction to the ﬁeld.
Revised: for a more complete introduction to the ﬁeld.

Original: Term-frequency based information retrieval.
Revised: Term-frequency based information retrieval.

Original: Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary.
Revised: Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary.

Original: Different weighing schemes have been proposed, to determine how important a particular term is to a document in a large dataset.
Revised: Different weighing schemes have been proposed, to determine how important a particular term is to a document in a large dataset.

Original: One of the most used weighing scheme is known as TF-IDF, and is based on inverse document frequency, or term speciﬁcity (Jones, 1972).
Revised: One of the most used weighing scheme is known as TF-IDF, and is based on inverse document frequency, or term speciﬁcity (Jones, 1972).

Original: BM25, which is still widely used today, extends TF-IDF (Robertson et al., 1995).
Revised: BM25, which is still widely used today, extends TF-IDF (Robertson et al., 1995).

Original: A well known limitation of these approaches is that they rely on exact match to retrieve documents.
Revised: A well known limitation of these approaches is that they rely on exact match to retrieve documents.

Original: This lead to the introduction of latent semantic analysis (Deerwester et al., 1990), in which documents are represented as low dimensional dense vectors.
Revised: This lead to the introduction of latent semantic analysis (Deerwester et al., 1990), in which documents are represented as low dimensional dense vectors.

Original: Neural network based information retrieval.
Revised: Neural network based information retrieval.

Original: Following the successful application of deep learning methods to natural language processing, neural networks techniques were introduced for information retrieval.
Revised: Following the successful application of deep learning methods to natural language processing, neural networks techniques were introduced for information retrieval.

Original: Huang et al.
Revised: Huang et al.

Original: (2013) proposed a deep bag-of-words model, in which representations of queries and documents are computed independently.
Revised: (2013) proposed a deep bag-of-words model, in which representations of queries and documents are computed independently.

Original: A relevance score is then obtained by taking the dot product between representations, and the model is trained end-to-end on click data from a search engine.
Revised: A relevance score is then obtained by taking the dot product between representations, and the model is trained end-to-end on click data from a search engine.

Original: This method was later reﬁned by replacing the bag-of-words model by convolutional neural networks (Shen et al., 2014) or recurrent neural network (Palangi et al., 2016).
Revised: This method was later reﬁned by replacing the bag-of-words model by convolutional neural networks (Shen et al., 2014) or recurrent neural network (Palangi et al., 2016).

Original: A limitation of bi-encoders is that queries and documents are represented by a single vector, preventing the model to capture ﬁne-grained interactions between terms.
Revised: A limitation of bi-encoders is that queries and documents are represented by a single vector, preventing the model to capture ﬁne-grained interactions between terms.

Original: Nogueira & Cho (2019) thus introduced a cross-encoder model, based on a pre-trained BERT model (Devlin et al., 2019), which jointly encodes queries and documents.
Revised: Nogueira & Cho (2019) thus introduced a cross-encoder model, based on a pre-trained BERT model (Devlin et al., 2019), which jointly encodes queries and documents.

Original: The application of a strong pre-trained model, as well as the cross-encoder architecture, lead to important improvement on the MS-MARCO benchmark (Bajaj et al., 2016).
Revised: The application of a strong pre-trained model, as well as the cross-encoder architecture, lead to important improvement on the MS-MARCO benchmark (Bajaj et al., 2016).

Original: The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with a traditional IR system such as BM25.
Revised: The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with a traditional IR system such as BM25.

Original: Gillick et al.
Revised: Gillick et al.

Original: ﬁrst studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking.
Revised: ﬁrst studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking.

Original: In the context of question answering, Karpukhin et al.
Revised: In the context of question answering, Karpukhin et al.

Original: (2020) introduced a dense passage retriever (DPR) based on the bi-encoder architecture.
Revised: (2020) introduced a dense passage retriever (DPR) based on the bi-encoder architecture.

Original: This model is initialized with a BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from
Revised: This model is initialized with a BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from

Original: BM25.
Revised: BM25.

Original: Xiong et al.
Revised: Xiong et al.

Original: (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS-MARCO dataset.
Revised: (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS-MARCO dataset.

Original: Once a collection of documents, such as Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al.
Revised: Once a collection of documents, such as Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al.

Original: To alleviate the limitations of bi-encoders, Humeau et al.
Revised: To alleviate the limitations of bi-encoders, Humeau et al.

Original: (2019) introduces the poly-encoder architecture, where documents are encoded by multiple vectors.
Revised: (2019) introduces the poly-encoder architecture, where documents are encoded by multiple vectors.

Original: Similarly, Khattab et al.
Revised: Similarly, Khattab et al.

Original: (2020) proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents.
Revised: (2020) proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents.

Original: To make the retrieval tractable, the term-level function is approximated to ﬁrst retrieve an initial set of candidates, which are then re-ranked with the true score.
Revised: To make the retrieval tractable, the term-level function is approximated to ﬁrst retrieve an initial set of candidates, which are then re-ranked with the true score.

Original: In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard &
Revised: In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard &

Original: Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020).
Revised: Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020).

Original: Luan et al.
Revised: Luan et al.

Original: (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders.
Revised: (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders.

Original: Dense retrievers, such as DPR, can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia.
Revised: Dense retrievers, such as DPR, can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia.

Original: Izacard et al.
Revised: Izacard et al.

Original: (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.
Revised: (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.

Original: Self-supervised learning for NLP.
Revised: Self-supervised learning for NLP.

Original: Following the success of word2vec (Mikolov et al., 2013), many self-supervised techniques have been proposed to learn representation of text.
Revised: Following the success of word2vec (Mikolov et al., 2013), many self-supervised techniques have been proposed to learn representation of text.

Original: Here, we brieﬂy review the ones that are most related to our approach: sentence level models and contrastive techniques.
Revised: Here, we brieﬂy review the ones that are most related to our approach: sentence level models and contrastive techniques.

Original: Jernite et al.
Revised: Jernite et al.

Original: (2017) introduced different objective function to learn sentence representations, including next sentence prediction and binary order prediction.
Revised: (2017) introduced different objective function to learn sentence representations, including next sentence prediction and binary order prediction.

Original: These objectives were later used in pre-trained models based on transformers, such as BERT (Devlin et al., 2019) and AlBERT (Lan et al., 2019).
Revised: These objectives were later used in pre-trained models based on transformers, such as BERT (Devlin et al., 2019) and AlBERT (Lan et al., 2019).

Original: In the context of retrieval, Lee et al.
Revised: In the context of retrieval, Lee et al.

Original: (2019) introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding a span of text.
Revised: (2019) introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding a span of text.

Original: Guu et al.
Revised: Guu et al.

Original: (2020) integrated a bi-encoder retriever model in a BERT pre-training scheme.
Revised: (2020) integrated a bi-encoder retriever model in a BERT pre-training scheme.

Original: The retrieved documents are used as additional context in the
Revised: The retrieved documents are used as additional context in the

Original: BERT task, and the whole system is trained end-to-end in an unsupervised way.
Revised: BERT task, and the whole system is trained end-to-end in an unsupervised way.

Original: Similarly, Lewis et al.
Revised: Similarly, Lewis et al.

Original: (2020) proposed to jointly learn a retriever and a generative seq2seq model, using self-supervised training.
Revised: (2020) proposed to jointly learn a retriever and a generative seq2seq model, using self-supervised training.

Original: Chang et al.
Revised: Chang et al.

Original: (2020) compares different pre-training tasks for retrieval, including the inverse cloze task.
Revised: (2020) compares different pre-training tasks for retrieval, including the inverse cloze task.

Original: Constrastive learning was introduced in computer vision by Wu et al.
Revised: Constrastive learning was introduced in computer vision by Wu et al.

Original: (2018), followed by several modiﬁcations to improve the training (He et al., 2020;
Revised: (2018), followed by several modiﬁcations to improve the training (He et al., 2020;

Original: Chen et al., 2020; Caron et al., 2020).
Revised: Chen et al., 2020; Caron et al., 2020).

Original: In the context of natural language processing, Fang et al.
Revised: In the context of natural language processing, Fang et al.

Original: (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation.
Revised: (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation.

Original: Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020;
Revised: Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020;

Original: Meng et al., 2021).
Revised: Meng et al., 2021).

Original: Finally, SBERT (Reimers & Gurevych, 2019) uses a siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings.
Revised: Finally, SBERT (Reimers & Gurevych, 2019) uses a siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings.

Original: Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora.
Revised: Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora.

Original: Concurrent to this work, Gao & Callan (2021) have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure.
Revised: Concurrent to this work, Gao & Callan (2021) have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we describe how to train a dense retriever with no supervision.
Revised: In this section, we describe how to train a dense retriever with no supervision.

Original: We review the model architecture and then describe contrastive learning - a key component of its training.
Revised: We review the model architecture and then describe contrastive learning - a key component of its training.

Original: The objective of a retriever is to ﬁnd relevant documents in a large collection for a given query.
Revised: The objective of a retriever is to ﬁnd relevant documents in a large collection for a given query.

Original: Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document.
Revised: Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document.

Original: A standard approach is to encode each query–document pair with a neural network (Nogueira & Cho, 2019).
Revised: A standard approach is to encode each query–document pair with a neural network (Nogueira & Cho, 2019).

Original: This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents.
Revised: This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents.

Original: Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently (Huang et al., 2013; Karpukhin et al., 2020).
Revised: Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently (Huang et al., 2013; Karpukhin et al., 2020).

Original: One can compute the relevance score by taking the dot product (or cosine similarity) between the document’s representation and the representation of the query.
Revised: One can compute the relevance score by taking the dot product (or cosine similarity) between the document’s representation and the representation of the query.

Original: More precisely, given a pair of query q and document d , we encode each of them independently using the same model, f θ , parametrized by θ .
Revised: More precisely, given a pair of query q and document d , we encode each of them independently using the same model, f θ , parametrized by θ .

Original: The relevance score s ( q, d ) is then the dot product of the resulting representations:
Revised: The relevance score s ( q, d ) is then the dot product of the resulting representations:

Original: [Equation]
Revised: [Equation]

Original: In practice, we use a transformer network for f θ to embed both queries and documents.
Revised: In practice, we use a transformer network for f θ to embed both queries and documents.

Original: The representation f θ ( q ) (resp.
Revised: The representation f θ ( q ) (resp.

Original: f θ ( d ) ) for a query (resp. document) is obtained by averaging the hidden representations of the last layer.
Revised: f θ ( d ) ) for a query (resp. document) is obtained by averaging the hidden representations of the last layer.

Original: Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to Devlin et al.
Revised: Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to Devlin et al.

Original: for more details.
Revised: for more details.

Original: 3. U NSUPERVISED TRAINING ON UNALIGNED DOCUMENTS
Revised: 3. U NSUPERVISED TRAINING ON UNALIGNED DOCUMENTS

Original: In this section, we describe our unsupervised training pipeline.
Revised: In this section, we describe our unsupervised training pipeline.

Original: We brieﬂy review the loss function traditionally used in contrastive learning.
Revised: We brieﬂy review the loss function traditionally used in contrastive learning.

Original: We then discuss obtaining positive pairs from a single text document, a critical ingredient for this training paradigm.
Revised: We then discuss obtaining positive pairs from a single text document, a critical ingredient for this training paradigm.

Original: 3.1. C ONTRASTIVE LEARNING
Revised: 3.1. C ONTRASTIVE LEARNING

Original: Contrastive learning is an approach that relies on the fact that every document is, in some way, unique.
Revised: Contrastive learning is an approach that relies on the fact that every document is, in some way, unique.

Original: This signal is the only information available in the absence of manual supervision.
Revised: This signal is the only information available in the absence of manual supervision.

Original: The resulting algorithm learns by discriminating between documents, using a contrastive loss (Wu et al., 2018).
Revised: The resulting algorithm learns by discriminating between documents, using a contrastive loss (Wu et al., 2018).

Original: This loss compares either positive (from the same document) or negative (from different documents) pairs of document representations.
Revised: This loss compares either positive (from the same document) or negative (from different documents) pairs of document representations.

Original: More formally, given a positive pair of representations ( q, k + ) and a set of negative pairs ( q, k i ) i =

Revised: More formally, given a positive pair of representations ( q, k + ) and a set of negative pairs ( q, k i ) i =

Original: K , the contrastive InfoNCE loss is deﬁned as:
Revised: K , the contrastive InfoNCE loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature parameter.
Revised: where τ is a temperature parameter.

Original: This loss encourages the relevance score of similar examples to be high and that of dissimilar examples to be low.
Revised: This loss encourages the relevance score of similar examples to be high and that of dissimilar examples to be low.

Original: Another interpretation of this loss function is the following: given the query representation q , the goal is to recover, or retrieve, the representation k + corresponding to the positive document, among all the negatives k i .
Revised: Another interpretation of this loss function is the following: given the query representation q , the goal is to recover, or retrieve, the representation k + corresponding to the positive document, among all the negatives k i .

Original: In the following, we refer to the left-hand side representations in the score s as queries and the right-hand side representations as keys.
Revised: In the following, we refer to the left-hand side representations in the score s as queries and the right-hand side representations as keys.

Original: 3.1. B UILDING POSITIVE PAIRS FROM A SINGLE DOCUMENT
Revised: 3.1. B UILDING POSITIVE PAIRS FROM A SINGLE DOCUMENT

Original: A crucial element of contrastive learning is how to build positive pairs from a single input.
Revised: A crucial element of contrastive learning is how to build positive pairs from a single input.

Original: In computer vision, this step relies on applying two independent data augmentations to the same image, resulting in two “views” that form a positive pair.
Revised: In computer vision, this step relies on applying two independent data augmentations to the same image, resulting in two “views” that form a positive pair (Wu et al., 2018; Chen et al., 2020).

Original: While we consider similar independent text transformation, we also explore dependent transformations designed to reduce the correlation between views.
Revised: While we consider similar independent text transformation, we also explore dependent transformations designed to reduce the correlation between views.

Original: Inverse Cloze Task. The inverse cloze task is a data augmentation that generates two mutually exclusive views of a document, introduced in the context of retrieval by Lee et al.
Revised: Inverse Cloze Task is a data augmentation that generates two mutually exclusive views of a document, introduced in the context of retrieval by Lee et al.

Original: The idea is to take a span of tokens to form one view, and its complementary to form the other view.
Revised: The idea is to take a span of tokens to form one view, and its complementary to form the other view.

Original: More precisely, given a sequence of text ( w 1 , ..., w n ) , a ﬁrst option is to sample a span a, b , where 1 ≤ a < b ≤ n , and to use the tokens of the span as the query and the rest of the tokens as the document (or key).
Revised: More precisely, given a sequence of text ( w 1 , ..., w n ) , a ﬁrst option is to sample a span a, b , where 1 ≤ a < b ≤ n , and to use the tokens of the span as the query and the rest of the tokens as the document (or key).

Original: Hence, in ICT, the two views are not generated independently to avoid overlapping.
Revised: In the implementation of Lee et al.

Original: Independent cropping.
Revised: the span considered as a query is a sentence. To encourage lexical matching, Lee et al. (2019) proposed to keep the span corresponding to the query in the document 10% of the time, leading to overlap.

Original: Independent cropping is a common independent data augmentation used for images where views are generated independently by cropping the input.
Revised: Independent cropping is a common independent data augmentation used for images where views are generated independently by cropping the input.

Original: In the context of text, cropping is equivalent to sampling a span of tokens.
Revised: In the context of text, cropping is equivalent to sampling a span of tokens.

Original: This strategy thus samples independently two spans from a document to form a positive pair.
Revised: This strategy thus samples independently two spans from a document to form a positive pair.

Original: As opposed to the inverse Cloze task, in cropping both views of the example correspond to contiguous subsequence of the original data.
Revised: As opposed to the inverse Cloze task, in cropping both views of the example correspond to contiguous subsequence of the original data.

Original: A second, and more important, difference between cropping and ICT, is the fact thatone can lead to overlap between the two views of the data, while the other cannot.
Revised: A second difference between cropping and ICT is the fact that the task is symmetric: both the queries and documents follow the same distribution.

Original: We argue that this overlap is beneﬁcial to retrieval pre-training by encouraging the network to ﬁnd exact matches between the query and the documents, in a way that is similar to lexical matching methods, like BM25.
Revised: Independent cropping can also lead to overlap between the two views of the data, hence encouraging the network to learn exact matches between the query and document, in a way that is similar to lexical matching methods like BM25.

Original: In practice, we can either ﬁx the length of the span for the query and the key, or sample them.
Revised: In practice, we can either ﬁx the length of the span for the query and the key, or sample them.

Original: Additional data augmentation.
Revised: Additional data augmentation.

Original: Finally, we also consider additional data augmentations such as random word deletion, random word replacement or random word masking.
Revised: Finally, we also consider additional data augmentations such as random word deletion, random word replacement or random word masking.

Original: We use these perturbation in addition to random cropping.
Revised: We use these perturbation in addition to random cropping.

Original: 3.1. B UILDING LARGE SET OF NEGATIVE PAIRS
Revised: 3.1. B UILDING LARGE SET OF NEGATIVE PAIRS

Original: An important aspect of contrastive learning is to maintain a large set of negative pairs.
Revised: An important aspect of contrastive learning is to maintain a large set of negative pairs.

Original: Most standard frameworks differ from how the negatives are handled, and we brieﬂy describe two of them, SimCLR and MoCo, that we use in this work.
Revised: Most standard frameworks differ from how the negatives are handled, and we brieﬂy describe two of them, SimCLR and MoCo, that we use in this work.

Original: Negative pairs within a batch.
Revised: Negative pairs within a batch.

Original: A ﬁrst solution, called SimCLR (Chen et al., 2020), is to generate the negative pairs by using the other examples from the same batch: each example in a batch is transformed twice to generate positive pairs, and we generate negative pairs by using the views from the other examples in the batch.
Revised: A ﬁrst solution, called SimCLR (Chen et al., 2020), is to generate the negative pairs by using the other examples from the same batch: each example in a batch is transformed twice to generate positive pairs, and we generate negative pairs by using the views from the other examples in the batch.

Original: In that case, the gradient is back-propagated through the representations of both the queries and the keys.
Revised: In that case, the gradient is back-propagated through the representations of both the queries and the keys.

Original: A downside of this approach is that it requires extremely large batch sizes to work well.
Revised: A downside of this approach is that it requires extremely large batch sizes to work well.

Revised: This technique was also considered when using ICT to pre-train retrievers by Lee et al.

Original: Negative pairs across batches.
Revised: Negative pairs across batches.

Original: An alternative approach is to store representations from previous batches in a queue and use them as negative examples in the loss (Wu et al., 2018).
Revised: An alternative approach is to store representations from previous batches in a queue and use them as negative examples in the loss (Wu et al., 2018).

Original: This allows for smaller batch size but slightly changes the loss by making it asymmetric between “queries” (one of the view generated from the elements of the current batch), and “keys” (the other generated views, as well as the elements stored in the queue).
Revised: This allows for smaller batch size but slightly changes the loss by making it asymmetric between “queries” (one of the view generated from the elements of the current batch), and “keys” (the other generated views, as well as the elements stored in the queue).

Original: Gradient is only backpropagated through the “queries”, and the representation of the “keys” are considered as ﬁxed.
Revised: Gradient is only backpropagated through the “queries”, and the representation of the “keys” are considered as ﬁxed.

Original: In practice, the features stored in the queue from previous batches comes form previous iterations of the network.
Revised: In practice, the features stored in the queue from previous batches comes form previous iterations of the network.

Original: This leads to a drop of performance when the network rapidly changes during training.
Revised: This leads to a drop of performance when the network rapidly changes during training.

Original: Instead,
Revised: Instead,

Original: He et al.
Revised: He et al.

Original: (2020) proposed to generate representations of keys from a second network that is updated more slowly.
Revised: (2020) proposed to generate representations of keys from a second network that is updated more slowly.

Original: This approach, called MoCo, considers two networks: one for the keys, parametrized by θ k , and one of the query, parametrized by θ q .
Revised: This approach, called MoCo, considers two networks: one for the keys, parametrized by θ k , and one of the query, parametrized by θ q .

Original: The parameters of the query network are updated with backpropagation and stochastic gradient descent, like in SimCLR, while the parameters of the key network, or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:
Revised: The parameters of the query network are updated with backpropagation and stochastic gradient descent, like in SimCLR, while the parameters of the key network, or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:

Original: [Equation]
Revised: [Equation]

Original: where m is the momentum parameter that takes its value in [0 , 1] .
Revised: where m is the momentum parameter that takes its value in [0 , 1] .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically evaluate contrastive learning as a way to train dense retrievers.
Revised: In this section, we empirically evaluate contrastive learning as a way to train dense retrievers.

Original: First, we compare our best models to the state-of-the-art on competitive retrieval benchmarks for both zero-shot and few-shot retrieval.
Revised: First, we compare our best models to the state-of-the-art on competitive retrieval benchmarks for both zero-shot and few-shot retrieval.

Original: Then, we provide an ablation study to motivate the technical choices leading to our best retriever, shedding some light on the important ingredients of our approach.
Revised: Then, we provide an ablation study to motivate the technical choices leading to our best retriever, shedding some light on the important ingredients of our approach.

Original: We give technical details about our models in Appendix A.
Revised: We give technical details about our models in Appendix A.

Original: 4. BEIR BENCHMARK
Revised: 4. BEIR BENCHMARK

Original: The BEIR benchmark, introduced by Thakur et al. (2021), contains 18 retrieval datasets with a focus on diversity.
Revised: The BEIR benchmark, introduced by Thakur et al. (2021), contains 18 retrieval datasets with a focus on diversity.

Original: Each dataset is made of a set of queries, the corresponding relevant documents and a large collection of documents to retrieve from.
Revised: Each dataset is made of a set of queries, the corresponding relevant documents and a large collection of documents to retrieve from.

Original: These datasets correspond to nine different retrieval tasks, such as fact checking, question answering, entity retrieval or citation prediction.
Revised: These datasets correspond to nine different retrieval tasks, such as fact checking, question answering, entity retrieval or citation prediction.

Original: Moreover, the benchmark covers a large number of domains, such as Wikipedia, news articles, scientiﬁc publications or social media posts.
Revised: Moreover, the benchmark covers a large number of domains, such as Wikipedia, news articles, scientiﬁc publications or social media posts.

Original: Finally, there is also diversity in terms of documents and queries length, with queries ranging from 3 to 190 words and documents from 11 to 630 words.
Revised: Finally, there is also diversity in terms of documents and queries length, with queries ranging from 3 to 190 words and documents from 11 to 630 words.

Original: Most datasets from the BEIR benchmark do not contain a training set, and the focus of the benchmark is zero-shot retrieval .
Revised: Most datasets from the BEIR benchmark do not contain a training set, and the focus of the benchmark is zero-shot retrieval .

Original: However, most machine learning based retrievers are still trained on supervised data, such as the large scale retrieval dataset MS-MARCO (Bajaj et al., 2016).
Revised: However, most machine learning based retrievers are still trained on supervised data, such as the large scale retrieval dataset MS-MARCO (Bajaj et al., 2016).

Original: Following standard practice, we report two metrics on this benchmark, the nDGC@10 and the Recall@100.
Revised: Following standard practice, we report two metrics on this benchmark, the nDGC@10 and the Recall@100.

Original: These two metrics are complementary and both important: nDCG focuses on the ranking of the top 10 retrieved documents, and is good at evaluating documents returned to human, for example in a search engine.
Revised: These two metrics are complementary and both important: nDCG focuses on the ranking of the top 10 retrieved documents, and is good at evaluating documents returned to human, for example in a search engine.

Original: On the other hand, the Recall@100 is relevant to evaluate retrievers that are included in marchine learning systems, such as question answering.
Revised: On the other hand, the Recall@100 is relevant to evaluate retrievers that are included in marchine learning systems, such as question answering.

Original: Indeed, such models can process hundreds of documents, and ignore the ranking of these documents (Izacard & Grave, 2020). 
Revised: Indeed, such models can process hundreds of documents, and ignore the ranking of these documents (Izacard & Grave, 2020). [Table]

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: First, we compare our approach to the BM25 ranking function, which does not require any training data.
Revised: First, we compare our approach to the BM25 ranking function, which does not require any training data.

Original: However, the relevance of individual terms depends on the inverse document frequency, which 
Revised: However, the relevance of individual terms depends on the inverse document frequency, which is computed over each document collection independently.

Original: [Table]
Revised: In that sense, the ranking function used in the BM25 baseline is adapted to each dataset on which retrieval is performed. We also compare to machine learning based retrievers, which can be classiﬁed in three categories: sparse, dense and late-interaction. For sparse methods, we compare to docT5query (Nogueira et al., 2019), which is a document expansion technique based on the T5 model (Raffel et al., 2019). The second sparse model we use as baseline is SPARTA (Zhao et al., 2020), which computes sparse representations of documents using a BERT model.

Original:  models trained on supervised data such as NaturalQuestions or MS-MARCO.
Revised: More precisely, the ranking function of SPARTA is based on interactions between terms of the document and the query. Because the representations of the query terms are non-contextualized, the scores can be pre-computed when indexing the database. For dense methods, we use DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2020) as baselines, which are bi-encoder models trained on supervised data such as NaturalQuestions or MS-MARCO.

Original: We also include the TAS-B technique (Hofstätter et al., 2021), which performs distillation from a cross-encoder to a bi-encoder model.
Revised: We also include the TAS-B technique (Hofstätter et al., 2021), which performs distillation from a cross-encoder to a bi-encoder model.

Original: Finally, we compare to GenQ, which uses a generative query model to create synthetic query-document pairs, and is applied to each dataset.
Revised: Finally, we compare to GenQ, which uses a generative query model to create synthetic query-document pairs, and is applied to each dataset.

Original: Thus, a different GenQ model is used for each dataset of the BEIR benchmark.
Revised: Thus, a different GenQ model is used for each dataset of the BEIR benchmark.

Original: For late-interaction methods, we use ColBERT (Khattab et al., 2020), which computes pairwise scores between contextualized representations of the query and the document.
Revised: For late-interaction methods, we use ColBERT (Khattab et al., 2020), which computes pairwise scores between contextualized representations of the query and the document.

Original: To make this approach scalable, this scoring function is only applied on documents obtained by applying an approximation of the full score.
Revised: To make this approach scalable, this scoring function is only applied on documents obtained by applying an approximation of the full score.

Original: Interestingly, it should be noted that BM is a very competitive baseline in the zero-shot setting, outperforming all dense retrievers on average over the BEIR benchmark.
Revised: Interestingly, it should be noted that BM is a very competitive baseline in the zero-shot setting, outperforming all dense retrievers on average over the BEIR benchmark.

Original: Only the re-ranking models and the query expansion (docT5query), which is also based on BM25, obtain better results than BM25.
Revised: Only the re-ranking models and the query expansion (docT5query), which is also based on BM25, obtain better results than BM25.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: In Table 1, we report the nDCG@10 on the BEIR benchmark for different retrievers.
Revised: In Table 1, we report the nDCG@10 on the BEIR benchmark for different retrievers.

Original: The corresponding Recall@100 performances can be found in Table 7 of appendix.
Revised: The corresponding Recall@100 performances can be found in Table 7 of appendix.

Original: We individually report results on each dataset as well as the average (“Avg.”) over 14 datasets of the BEIR Benchmark (excludingfor license reasons).
Revised: We individually report results on each dataset as well as the average (“Avg.”) over 14 datasets of the BEIR Benchmark (excludingfor license reasons).

Original: We omit retrievers that use a re-ranker during inference since it is orthogonal to our focus and can be applied to any method.
Revised: We omit retrievers that use a re-ranker during inference since it is orthogonal to our focus and can be applied to any method.

Original: We observe that contrastive pre-training leads to strong performance among retrievers: our model obtains state-of-the art performance when ﬁnetuned on the MS-MARCO dataset.

Original: It should be noted

Original: [Table]
Revised: [Table]

Original: nDCG@10 on the test sets of benchmarks when ﬁnetuned on a small training set.
Revised: nDCG@10 on the test sets of benchmarks when ﬁnetuned on a small training set.

Original: We compare
Revised: We compare

Original: BERT and our pre-training with and without the ﬁnetuning step on
Revised: BERT and our pre-training with and without the ﬁnetuning step on

Original: MS-MARCO.
Revised: MS-MARCO.

Original: Note that our unsupervised pre-training alone leads to better performance than BERT ﬁnetuned on MS-MARCO.
Revised: Note that our unsupervised pre-training alone leads to better performance than BERT ﬁnetuned on MS-MARCO.

Original: [Figure]
Revised: [Figure]

Revised: We observe that contrastive pre-training leads to strong performance among retrievers: our model obtains state-of-the art performance when ﬁnetuned on the MS-MARCO dataset.

Original:  that our ﬁnetuning procedure on MS-MARCO is simpler than for other dense retrievers: we use a simple strategy for negative mining and do not use distillation.
Revised: It should be noted that our ﬁnetuning procedure on MS-MARCO is simpler than for other dense retrievers: we use a simple strategy for negative mining and do not use distillation.

Original: Our model would probably also beneﬁts from improvements proposed by these retrievers, but this is beyond the scope of this paper.
Revised: Our model would probably also beneﬁts from improvements proposed by these retrievers, but this is beyond the scope of this paper.

Original: We also illustrate the beneﬁt of our dense retriever compared to BM25 by evaluating it in a few shot setting, where we have access to a small number of training queries and relevant documents to ﬁnetune locally the retriever.
Revised: We also illustrate the beneﬁt of our dense retriever compared to BM25 by evaluating it in a few shot setting, where we have access to a small number of training queries and relevant documents to ﬁnetune locally the retriever.

Original: This setting is common in practice, and lexical based methods, like
Revised: This setting is common in practice, and lexical based methods, like

Original: BM25, cannot beneﬁt leverage the small training sets to adapt its weights.
Revised: BM25, cannot beneﬁt leverage the small training sets to adapt its weights.

Original: In Table 2, we report the nDCG@10 results on the three datasets from the BEIR benchmark associated with the smallest training sets.
Revised: In Table 2, we report the nDCG@10 results on the three datasets from the BEIR benchmark associated with the smallest training sets.

Original: These training sets range from 809 to almost 5,500 queries.
Revised: These training sets range from 809 to almost 5,500 queries.

Original: We observe that on these small datasets, our pre-training leads to signiﬁcantly better results than BERT pre-training only, even when BERT model is ﬁnetuned on MS-MARCO.
Revised: We observe that on these small datasets, our pre-training leads to signiﬁcantly better results than BERT pre-training only, even when BERT model is ﬁnetuned on MS-MARCO.

Original: Our pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the low-shot setting.
Revised: Our pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the low-shot setting.

Original: Finally, we compare the performance of fully unsupervised models, i.e., without ﬁnetuning on
Revised: Finally, we compare the performance of fully unsupervised models, i.e., without ﬁnetuning on

Original: MS-MARCO or other annotated data.
Revised: MS-MARCO or other annotated data.

Original: In Table 8 we report the retrieval performance of our model on two standard question answering benchmarks: NaturalQuestions (Kwiatkowski et al., 2019) and
Revised: In Table 8 we report the retrieval performance of our model on two standard question answering benchmarks: NaturalQuestions (Kwiatkowski et al., 2019) and

Original: TriviaQA (Joshi et al., 2017).
Revised: TriviaQA (Joshi et al., 2017).

Original: In Figure 1 we report the performance of unsupervised models on the
Revised: In Figure 1 we report the performance of unsupervised models on the

Original: BEIR benchmark.
Revised: BEIR benchmark.

Original: Interestingly, we observe that, in this setting, our pre-training can lead to a model with competitive retrieval performance compared to BM25.
Revised: Interestingly, we observe that, in this setting, our pre-training can lead to a model with competitive retrieval performance compared to BM25.

Original: Our approach outperforms previously proposed unsupervised dense retrievers.
Revised: Our approach outperforms previously proposed unsupervised dense retrievers.

Original: On question answering benchmarks it also outperforms a strong BM25 baseline (Ma et al., 2021) by 3% for the Recall@100, and is on par with BM25 on
Revised: On question answering benchmarks it also outperforms a strong BM25 baseline (Ma et al., 2021) by 3% for the Recall@100, and is on par with BM25 on

Original: TriviaQA for the Recall@100.
Revised: TriviaQA for the Recall@100.

Original: On datasets of the BEIRbenchmark, we observe that our unsupervised retriever is competitive with BM25 on every datasets, but TREC-COVID and Tóuche-2020.
Revised: On datasets of the BEIRbenchmark, we observe that our unsupervised retriever is competitive with BM25 on every datasets, but TREC-COVID and Tóuche-2020.

Original: These results show the potential of contrastive learning to develop fully unsupervised dense retrievers.
Revised: These results show the potential of contrastive learning to develop fully unsupervised dense retrievers.

Original:  A BLATION STUDIES
Revised:  A BLATION STUDIES

Original: In this section, we investigate the inﬂuence of different design choices on our method.
Revised: In this section, we investigate the inﬂuence of different design choices on our method.

Original: In these ablations, all the models are pre-trained on Wikipedia for 200 k gradient steps, with a batch size of 2,048 (on 32 GPUs).
Revised: In these ablations, all the models are pre-trained on Wikipedia for 200 k gradient steps, with a batch size of 2,048 (on 32 GPUs).

Original: Each ﬁne-tuning on MS-MARCO takes 20 k gradient steps with a batch size of 512 (on 8 GPUs), using AdamW and no hard negatives.
Revised: Each ﬁne-tuning on MS-MARCO takes 20 k gradient steps with a batch size of 512 (on 8 GPUs), using AdamW and no hard negatives.

Original: MoCo vs. SimCLR.
Revised: MoCo vs. SimCLR.

Original: First, we compare the two contrastive pre-training methods: MoCo and
Revised: First, we compare the two contrastive pre-training methods: MoCo and

Original: SimCLR.
Revised: SimCLR.

Original: As in SimCLR, the number of negative examples is equal to the batch size, we train models with a batch size of 4,096 and restrict the queue in MoCo to the same number of elements.
Revised: As in SimCLR, the number of negative examples is equal to the batch size, we train models with a batch size of 4,096 and restrict the queue in MoCo to the same number of elements.

Original: This experiment measures the effect of using of momentum encoder for the keys instead of the same network as for the queries.
Revised: This experiment measures the effect of using of momentum encoder for the keys instead of the same network as for the queries.

Original: Using a momentum also prevents from backpropagating the gradient through the keys.
Revised: Using a momentum also prevents from backpropagating the gradient through the keys.

Original: We report results, with ﬁne-tuning on MS-MARCO in Table 5.
Revised: We report results, with ﬁne-tuning on MS-MARCO in Table 5.

Original: We observe that the difference of performance between the two methods is small, especially after ﬁne-tuning on
Revised: We observe that the difference of performance between the two methods is small, especially after ﬁne-tuning on

Original: MS-MARCO.
Revised: MS-MARCO.

Original: This validates our choice of MoCo as our contrastive learning framework, since it scales to a larger number of negative examples without the need to increase the batch size.
Revised: This validates our choice of MoCo as our contrastive learning framework, since it scales to a larger number of negative examples without the need to increase the batch size.

Original: [Table]
Revised: [Table]

Original: Number of negative examples.
Revised: Number of negative examples.

Original: Next, we study the inﬂuence of the number of negatives used in the contrastive loss, by varying the queue size of the MoCo algorithm.
Revised: Next, we study the inﬂuence of the number of negatives used in the contrastive loss, by varying the queue size of the MoCo algorithm.

Original: We consider values ranging from 2,048 to 131,072, and report results in Figure 2.
Revised: We consider values ranging from 2,048 to 131,072, and report results in Figure 2.

Original: We see that on average over the BEIR benchmark, increasing the number of negatives leads to better retrieval performance, especially in the unsupervised setting.
Revised: We see that on average over the BEIR benchmark, increasing the number of negatives leads to better retrieval performance, especially in the unsupervised setting.

Original: However, we note that this effect is not equally strong for all datasets.
Revised: However, we note that this effect is not equally strong for all datasets [Figure]

Revised: framework where the keys for the negatives are computed with the momentum encoder and stored in a queue.

Original: Data augmentations.
Revised: Data augmentations.

Original: Third, we compare different ways to generate pairs of positive examples from a single document or chunk of text.
Revised: Third, we compare different ways to generate pairs of positive examples from a single document or chunk of text.

Original: In particular, we compare random cropping, which leads to pairs with overlap, and the inverse cloze task, which was previously considered to pre-train retrievers.
Revised: In particular, we compare random cropping, which leads to pairs with overlap, and the inverse cloze task, which was previously considered to pre-train retrievers.

Original: Interestingly, as shown in Table 4, the random cropping strategy strongly outperforms the inverse cloze task in our setting.
Revised: Interestingly, as shown in Table 4, the random cropping strategy strongly outperforms the inverse cloze task in our setting.

Original: We argue that overlap between the two views of the data is beneﬁcial, as it allows the network to learn lexical matching, similarly to BM25.
Revised: We argue that overlap between the two views of the data is beneﬁcial, as it allows the network to learn lexical matching, similarly to BM25.

Original: We also investigate whether additional data perturbations, such as random word deletion or replacement, are beneﬁcial for retrieval.
Revised: We also investigate whether additional data perturbations, such as random word deletion or replacement, are beneﬁcial for retrieval.

Original: After ﬁne-tuning on MS-MARCO, it seems that such data perturbations slightly degrade the performance of dense retrievers.
Revised: After ﬁne-tuning on MS-MARCO, it seems that such data perturbations slightly degrade the performance of dense retrievers

Original: [Table]
Revised: [Table]

Original: Training data.
Revised: Training data.

Original: Finally, we study the impact of the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet or a mix of both sources of data.
Revised: Finally, we study the impact of the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet or a mix of both sources of data.

Original: We report results in
Revised: We report results in

Original: Table 5, and observe that there is no clear winner between the two data sources.
Revised: Table 5, and observe that there is no clear winner between the two data sources.

Original: Unsurprisingly, pre-training on Wikipedia leads to better performance on datasets for which documents also come from Wikipedia, such as NaturalQuestions or FEVER.
Revised: Unsurprisingly, pre-training on Wikipedia leads to better performance on datasets for which documents also come from Wikipedia, such as NaturalQuestions or FEVER.

Original: On the other hand, on datasets from different[Figure] domains than Wikipedia such as FiQA or ArguAna, training on the more diverse CCNet data leads to better results.
Revised: On the other hand, on datasets from different domains than Wikipedia such as FiQA or ArguAna, training on the more diverse CCNet data leads to better results.

Original: To get the best of both worlds, we consider to strategies to mix the two data sources.
Revised: To get the best of both worlds, we consider to strategies to mix the two data sources.

Original: In the “50/50%” strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet.
Revised: In the “50/50%” strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet.

Original: In the “uniform” strategy, examples are sampled uniformly over the union of the dataset.
Revised: In the “uniform” strategy, examples are sampled uniformly over the union of the dataset.

Original: Since CCNet is signiﬁcantly larger than Wikipedia, this means that most of the batches are from CCNet.
Revised: Since CCNet is signiﬁcantly larger than Wikipedia, this means that most of the batches are from CCNet.

Original: [Table]
Revised: [Table]

Original: Impact of ﬁne-tuning on MS-MARCO.
Revised: Impact of ﬁne-tuning on MS-MARCO.

Original: To isolate the impact of our pre-training method as opposed to the impact of our ﬁne-tuning on MS-MARCO, we propose to apply the same ﬁne-tuning to the BERT base uncased model.
Revised: To isolate the impact of our pre-training method as opposed to the impact of our ﬁne-tuning on MS-MARCO, we propose to apply the same ﬁne-tuning to the BERT base uncased model.

Original: We report results in Table 6, and observe that when applied to
Revised: We report results in Table [Table] and observe that when applied to

Original: BERT, our ﬁne-tuning leads to results that are lower than the state-of-the-art.
Revised: BERT, our ﬁne-tuning leads to results that are lower than the state-of-the-art.

Original: Hence, we believe that most of the improvement compared to the state-of-the-art retrievers can be attributed to our contrastive pre-training strategy. 
Revised: Hence, we believe that most of the improvement compared to the state-of-the-art retrievers can be attributed to our contrastive pre-training strategy. [Table]

Original: D ISCUSSION
Revised: D ISCUSSION

Original: In this work, we propose to use contrastive pre-training to learn dense text retrievers.
Revised: In this work, we propose to explore the limits of contrastive pre-training to learn dense text retrievers.

Original: We use the Momentum Contrast, or MoCo, technique, which allows us to train with a large number of negative examples.
Revised: We use the MoCo technique, which allows us to train with a large number of negative examples.

Original: We make several interesting observations:
Revised: We make several interesting observations:

Original: ﬁrst, while the inverse Cloze task has been[Table] introduced in the context of , we show that independent random cropping leads to better results
Revised: ﬁrst, while the inverse Cloze task has been introduced in the context of retrieval, we show that independent random cropping leads to better results.

Original: We believe that a key difference between the two is the potential overlap between the two views of the document, allowing the network to capture lexical matching.

Original: Second, we show that neural networks trained without supervision using contrastive learning exhibits good (albeit not state-of-the-art) retrieval performance.
Revised: Second, we show that neural networks trained without supervision using contrastive learning exhibits good (albeit not state-of-the-art) retrieval performance.

Original: Based on these observations, we trained a model that obtains state-of-the-art performance on the BEIR benchmark when ﬁne-tuned on MS-MARCO.
Revised: Based on these observations, we trained a model that obtains state-of-the-art performance on the BEIR benchmark when ﬁne-tuned on MS-MARCO.

Original: A T ECHNICAL DETAILS
Revised: A T ECHNICAL DETAILS

Original: Contrastive pre
Revised: Contrastive pre

Original: training.
Revised: training.

Original: For the model with ﬁne-tuning on MS-MARCO (respectively, without ﬁne-tuning), we use the MoCo algorithm He et al.
Revised: For the model with ﬁne-tuning on MS-MARCO (respectively, without ﬁne-tuning), we use the MoCo algorithm He et al.

Original: (2020) with a queue of size 65536 (resp. 131,072), a momentum value of 0.999 (resp.
Revised: (2020) with a queue of size 65536 (resp. 131,072), a momentum value of 0.999 (resp.

Original: 0.9995) and a temperature of 0.05.
Revised: 0.9995) and a temperature of 0.05.

Original: We use the random cropping data augmentation, with documents of 256 tokens and span sizes sampled between 10% and 50% (resp.
Revised: We use the random cropping data augmentation, with documents of 256 tokens and span sizes sampled between 10% and 50% (resp.

Original: 5% and 50%) of the document length.
Revised: 5% and 50%) of the document length.

Original: We also apply word deletion with a probability of 20%.
Revised: We also apply word deletion with a probability of 20%.

Original: We optimize the model with the AdamW (Loshchilov & Hutter, 2019) optimizer, with learning rate of 10 − 4 (resp.
Revised: We optimize the model with the AdamW (Loshchilov & Hutter, 2019) optimizer, with learning rate of 10 − 4 (resp.

Original: 5 · 10 − 5 ), batch size of 2,048 and 500,000 (resp. 200,000) steps.
Revised: 5 · 10 − 5 ), batch size of 2,048 and 500,000 (resp. 200,000) steps.

Original: We initialize the network with the publicly available BERT base uncased model.
Revised: We initialize the network with the publicly available BERT base uncased model.

Original: For training data, we use a data from
Revised: For training data, we use a data from

Original: Wikipedia (resp.
Revised: Wikipedia (resp.

Original: a mix between Wikipedia and CC-net data (Wenzek et al., 2020), where half the batches are sampled from each source).
Revised: a mix between Wikipedia and CC-net data (Wenzek et al., 2020), where half the batches are sampled from each source).

Original: Fine-tuning on MS-MARCO.
Revised: Fine-tuning on MS-MARCO.

Original: When ﬁne-tuning our model on MS-MARCO, we use the ASAM optimizer (Kwon et al., 2021), with a learning rate of 10 − 5 and a batch size of 1024 with a temperature of 0 . 05 , also used during pretraining.
Revised: When ﬁne-tuning our model on MS-MARCO, we use the ASAM optimizer (Kwon et al., 2021), with a learning rate of 10 − 5 and a batch size of 1024 with a temperature of 0 . 05 , also used during pretraining.

Original: We train an initial model with random negative examples for 20000 steps, mine hard negatives with this ﬁrst model [Table] and re-train a second model with those.
Revised: We train an initial model with random negative examples for 20000 steps, mine hard negatives with this ﬁrst model, and re-train a second model with those.

Original: Each query is associated with a gold document and a negative document, which is a random document in the ﬁrst phase and a hard negative 10% of the time in the second phase.
Revised: Each query is associated with a gold document and a negative document, which is a random document in the ﬁrst phase and a hard negative 10% of the time in the second phase.

Original: All documents from a batch are used as negatives for each query
Revised: All documents from a batch are used as negatives for each query

Original: [Table]
Revised: [Table]

Original: [Table]