Original: R EVISITING D ENSE R ETRIEVAL WITH U NASWERABLE
Revised: R EVISITING D ENSE R ETRIEVAL WITH U NASWERABLE

Original: C OUNTERFACTUALS
Revised: C OUNTERFACTUALS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The retriever-reader framework is popular for open-domain question answering (ODQA), where a retriever samples for the reader a set of relevant candidate passages from a large corpus.
Revised: The retriever-reader framework is popular for open-domain question answering (ODQA), where a retriever samples for the reader a set of relevant candidate passages from a large corpus.

Original: A key assumption behind this method is that high relevance score from the retriever likely indicates high answerability from the reader, which implies a high probability that the retrieved passages contain answers to a given question.
Revised: A key assumption behind this method is that high relevance score from the retriever likely indicates high answerability from the reader, which implies a high probability that the retrieved passages contain answers to a given question.

Original: In this work, we empirically dispel this belief and observe that recent dense retrieval models based on DPR often rank unanswerable counterfactual passages higher than their answerable original passages.
Revised: In this work, we empirically dispel this belief and observe that recent dense retrieval models based on DPR often rank unanswerable counterfactual passages higher than their answerable original passages.

Original: To address such answer-unawareness in dense retrievers, we seek to use counterfactual samples as additional training resources to better synchronize the relevance measurement of DPR with the answerability of question-passage pairs.
Revised: To address such answer-unawareness in dense retrievers, we seek to use counterfactual samples as additional training resources to better synchronize the relevance measurement of DPR with the answerability of question-passage pairs.

Original: Specifically, we present counterfactually-
Revised: Specifically, we present counterfactually-

Original: Pi voting C ontrastive L earning (PiCL), a novel representation learning approach for passage retrieval that leverages counterfactual samples as pivots between positive and negative samples in their learned embedding space.
Revised: Pi voting C ontrastive L earning (PiCL), a novel representation learning approach for passage retrieval that leverages counterfactual samples as pivots between positive and negative samples in their learned embedding space.

Original: We incorporate PiCL into the retriever training to show the effectiveness of PiCL on ODQA benchmarks and the robustness of the learned models.
Revised: We incorporate PiCL into the retriever training to show the effectiveness of PiCL on ODQA benchmarks and the robustness of the learned models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Open-domain question answering (ODQA)
Revised: Open-domain question answering (ODQA)

Original: (Chen & Yih, 2020) is a task that finds the answers to natural language questions from a large collection of documents.
Revised: (Chen & Yih, 2020) is a task that finds the answers to natural language questions from a large collection of documents.

Original: A common approach to ODQA tasks is a two-stage retriever-reader framework (Chen et al., 2017), where a retriever roughly selects relevant candidate passages from which a reader extracts the answers to a question.
Revised: A common approach to ODQA tasks is a two-stage retriever-reader framework (Chen et al., 2017), where a retriever roughly selects relevant candidate passages from which a reader extracts the answers to a question.

Original: For the firststage retrieval, recent studies leverage dense passage retriever (DPR) (Karpukhin et al., 2020), which computes relevance scores based on the similarity between learned representations of questions and passages.
Revised: For the firststage retrieval, recent studies leverage dense passage retriever (DPR) (Karpukhin et al., 2020), which computes relevance scores based on the similarity between learned representations of questions and passages.

Original: Generally, it is assumed that high relevance scores from the retriever naturally lead to high answerability from the reader, which implies that the retrieved passages are more likely to contain the correct answer to a given question.
Revised: Generally, it is assumed that high relevance scores from the retriever naturally lead to high answerability from the reader, which implies that the retrieved passages are more likely to contain the correct answer to a given question.

Original: Based on such naive belief, many research efforts for ODQA have focused on solely improving the ranking performance of the retrievers to further enhance the performance of the subsequent reader (Karpukhin et al., 2020; Xiong et al., 2021;
Revised: Based on such naive belief, many research efforts for ODQA have focused on solely improving the ranking performance of the retrievers to further enhance the performance of the subsequent reader (Karpukhin et al., 2020; Xiong et al., 2021;

Original: Qu et al., 2021).
Revised: Qu et al., 2021).

Original: In this work, our counterfactual simulation dispels this myth.
Revised: In this work, our counterfactual simulation dispels this myth.

Original: Specifically, we present a simple counterfactual sampling strategy that removes the answer span of a question from its positive passage.
Revised: Specifically, we present a simple counterfactual sampling strategy that removes the answer span of a question from its positive passage.

Original: We observe that neural retrieval models based on DPR often rank such unanswerable counterfactual passages similarly to or higher than their answerable original passages.
Revised: We observe that neural retrieval models based on DPR often rank such unanswerable counterfactual passages similarly to or higher than their answerable original passages.

Original: We hypothesize that dense retrievers are not answer-aware, which poses the asynchronicity between their relevance scores and answerability as a new challenge of the current retriever-reader framework.
Revised: We hypothesize that dense retrievers are not answer-aware, which poses the asynchronicity between their relevance scores and answerability as a new challenge of the current retriever-reader framework.

Original: Motivated by this, we seek to repurpose the counterfactual passages as additional training resources to better synchronize the relevance measurement of DPR with the answerability of retrieved passages.
Revised: Motivated by this, we seek to repurpose the counterfactual passages as additional training resources to better synchronize the relevance measurement of DPR with the answerability of retrieved passages.

Original: A straightforward implementation is to consider these counterfactual samples as hard negatives to supplement in-batch negatives (Zhan et al., 2021).
Revised: A straightforward implementation is to consider these counterfactual samples as hard negatives to supplement in-batch negatives (Zhan et al., 2021).

Original: However, the semantic overlap between the counterfactual and source-original passages makes such naive application ineffective, as training a retriever with minimally different negative samples ( i.e. , answer-removed counterfactuals) potentially hurts the ability to capture semantic relevance of positive question-passage pairs.
Revised: However, the semantic overlap between the counterfactual and source-original passages makes such naive application ineffective, as training a retriever with minimally different negative samples ( i.e. , answer-removed counterfactuals) potentially hurts the ability to capture semantic relevance of positive question-passage pairs.

Original: To overcome this challenge, we propose counterfactually-Pivoting Contrastive Learning (PiCL), a novel contrastive representation learning scheme for improving DPR.
Revised: To overcome this challenge, we propose counterfactually-Pivoting Contrastive Learning (PiCL), a novel contrastive representation learning scheme for improving DPR.

Original: By PiCL, our ultimate goal is to make DPR be aware of both relevance and answerability on its dense embedding space, i.e. , higher relevance leads to higher answerability, and vice versa.
Revised: By PiCL, our ultimate goal is to make DPR be aware of both relevance and answerability on its dense embedding space, i.e. , higher relevance leads to higher answerability, and vice versa.

Original: For that, the key idea of PiCL is to use the counterfactual samples as not only hard negatives but also pseudo positives, which can be a pivot between the original positive and negative passages.
Revised: For that, the key idea of PiCL is to use the counterfactual samples as not only hard negatives but also pseudo positives, which can be a pivot between the original positive and negative passages.

Original: To effectively learn with such multiple views,
Revised: To effectively learn with such multiple views,

Original: PiCL aims to optimize the following three loss functions of learning the dense embeddings of question and original/counterfactual passages: (1) modified DPR loss of mapping the positive passage closer to the question than negatives (including counterfactuals) in a batch, (2) Counterfactuals-ashard-negatives loss of mapping the original positive passage closer to the question than counterfactuals, and (3) Counterfactuals-as-pseudo-positives loss of mapping the counterfactuals closer to the question than in-batch negative passages.
Revised: PiCL aims to optimize the following three loss functions of learning the dense embeddings of question and original/counterfactual passages: (1) modified DPR loss of mapping the positive passage closer to the question than negatives (including counterfactuals) in a batch, (2) Counterfactuals-ashard-negatives loss of mapping the original positive passage closer to the question than counterfactuals, and (3) Counterfactuals-as-pseudo-positives loss of mapping the counterfactuals closer to the question than in-batch negative passages.

Original: Our contributions are twofold.
Revised: Our contributions are twofold.

Original: First, we demonstrate that the neural retrieval models suffer from the unawareness of answerability, which can be a bottleneck of the retriever-reader pipeline for ODQA.
Revised: First, we demonstrate that the neural retrieval models suffer from the unawareness of answerability, which can be a bottleneck of the retriever-reader pipeline for ODQA.

Original: Second, we design a novel training framework named PiCL of learning to synchronize the relevance and answerability.
Revised: Second, we design a novel training framework named PiCL of learning to synchronize the relevance and answerability.

Original: Our extensive experiments validate the effectiveness of PiCL in diverse ODQA scenarios, which are orthogonally applicable to well-studied prior techniques on DPR.
Revised: Our extensive experiments validate the effectiveness of PiCL in diverse ODQA scenarios, which are orthogonally applicable to well-studied prior techniques on DPR.

Original:  P RELIMINARIES AND R ELATED WORK 2. D ENSE P ASSAGE R ETRIEVAL FOR ODQA
Revised:  P RELIMINARIES AND R ELATED WORK 2. D ENSE P ASSAGE R ETRIEVAL FOR ODQA

Original: Open-domain question answering is a knowledge-intensive task that aims at answering factoid questions given a large corpus of texts (Chen & Yih, 2020).
Revised: Open-domain question answering is a knowledge-intensive task that aims at answering factoid questions given a large corpus of texts (Chen & Yih, 2020).

Original: We particularly focus on a setting in which the corpus C is a collection of M passages p , i.e. C = { p i } Mi =1 .
Revised: We particularly focus on a setting in which the corpus C is a collection of M passages p , i.e. C = { p i } Mi =1 .

Original: For each question-passage pair ( q i , p + i ) where q i is answerable from p
Revised: For each question-passage pair ( q i , p + i ) where q i is answerable from p

Original: + i , the passage p
Revised: + i , the passage p

Original: + i can be viewed as a concatenation of nonanswer spans s l and s r and an answer span a i , i.e. p + i =
Revised: + i can be viewed as a concatenation of nonanswer spans s l and s r and an answer span a i , i.e. p + i =

Original: [ s l ; a i ; s r ] .
Revised: [ s l ; a i ; s r ] .

Original: While s l and s r provide some relevant contexts to the question, the answerability of q i from p i is determined by the presence of answer span a i , which contains not only the exact match for the gold answer but also key evidence to the question.
Revised: While s l and s r provide some relevant contexts to the question, the answerability of q i from p i is determined by the presence of answer span a i , which contains not only the exact match for the gold answer but also key evidence to the question.

Original: Given a question q
Revised: Given a question q

Original: i , the reader finds the corresponding answer span a i from an answerable passage p
Revised: i , the reader finds the corresponding answer span a i from an answerable passage p

Original: + i in the corpus C .
Revised: + i in the corpus C .

Original: Due to the large search space in the corpus C , a first-stage retriever is commonly used in ODQA to find subsets of relevant passages to questions for the expensive reader.
Revised: Due to the large search space in the corpus C , a first-stage retriever is commonly used in ODQA to find subsets of relevant passages to questions for the expensive reader.

Original: The predominant approach to passage retrieval is DPR (Karpukhin et al., 2020), which leverages the efficient dual-encoder architecture denoted as [ f q , f p ] to encode questions and passages into a learned embedding space.
Revised: The predominant approach to passage retrieval is DPR (Karpukhin et al., 2020), which leverages the efficient dual-encoder architecture denoted as [ f q , f p ] to encode questions and passages into a learned embedding space.

Original: For a question-answer passage pair ( q i , p + i ) and a set of N negative passages p − j , DPR is trained to maximize the similarity between the question q i and its answer passage p
Revised: For a question-answer passage pair ( q i , p + i ) and a set of N negative passages p − j , DPR is trained to maximize the similarity between the question q i and its answer passage p

Original: + i :
Revised: + i :

Original: [Equation]
Revised: [Equation]

Original: where sim ( · , · ) is a function that computes the relevance score between a question and a passage as dot product between the question embedding f q ( q i ) and the passage embedding f p ( p i ) :
Revised: where sim ( · , · ) is a function that computes the relevance score between a question and a passage as dot product between the question embedding f q ( q i ) and the passage embedding f p ( p i ) :

Original: [Equation]
Revised: [Equation]

Original: At runtime, the retriever indexes all passages based on the similarity metric and performs efficient search using approximate nearest neighbor search libraries (Johnson & Douze, 2019).
Revised: At runtime, the retriever indexes all passages based on the similarity metric and performs efficient search using approximate nearest neighbor search libraries (Johnson & Douze, 2019).

Original: To further enhance the discriminative power of DPR, more recent studies on dense retrieval adopt model-centric approaches (Xiong et al., 2021;
Revised: To further enhance the discriminative power of DPR, more recent studies on dense retrieval adopt model-centric approaches (Xiong et al., 2021;

Original: Qu et al., 2021; Izacard & Grave, 2021), which involve iterative training pipelines that exploit multiple fine-tuned encoders.
Revised: Qu et al., 2021; Izacard & Grave, 2021a), which involve iterative training pipelines that exploit multiple fine-tuned encoders.

Original: For example, state-ofthe-art models including ANCE (Xiong et al., 2021) and RocketQA (Qu et al., 2021) focus largely on combining multiple dual- and cross-encoders into sophisticated frameworks for hard negative sampling.
Revised: For example, state-ofthe-art models including ANCE (Xiong et al., 2021) and RocketQA (Qu et al., 2021) focus largely on combining multiple dual- and cross-encoders into sophisticated frameworks for hard negative sampling.

Original: While effective, as presented in Table1, using these complex model-centric approaches requires  significant amounts of computing resources, which limits their development in diverse environments (Lindgren et al., 2021; Du et al., 2022; Gao et al., 2022).
Revised: While effective, as presented in Table 1, using these complex model-centric approaches requires a significant amount of computing resources, which limits their development in diverse environments (Lindgren et al., 2021; Du et al., 2022; Gao et al., 2022).

Original: In this work, we instead focus on a data-centric approach, which aims to maximally leverage the given data sources without complicating the training process, to improve the ranking performance and robustness of DPR.
Revised: In this work, we instead focus on a data-centric approach, which aims to maximally leverage the given data sources without complicating the training process, to improve the ranking performance and robustness of DPR.

Original: We later show that our data-centric approach based on counterfactual augmentation is both effective as alternatives to DPR training and orthogonal to existing model-centric approaches.
Revised: We later show that our data-centric approach based on counterfactual augmentation is both effective as alternatives to DPR training and orthogonal to existing model-centric approaches.

Original: [Table]
Revised: [Table]

Original: C OUNTERFACTUAL L EARNING FOR T EXT D
Revised: C OUNTERFACTUAL L EARNING FOR T EXT D

Original: [Table]
Revised: [Table]

Original: Counterfactual inference has been applied to representation learning to obtain fair representations (Kusner et al., 2017) in various domains such as image classification (Goyal et al., 2019) and vision-language tasks (Liang et al., 2020; Niu et al., 2021).
Revised: Counterfactual inference has been applied to representation learning to obtain fair representations (Kusner et al., 2017) in various domains such as image classification (Goyal et al., 2019) and vision-language tasks (Liang et al., 2020; Niu et al., 2021).

Original: The key idea behind counterfactual learning is to train a model that is invariant to specific aspects of the input data (Johansson et al., 2016; Kusner et al., 2017)
Revised: The key idea behind counterfactual learning is to train a model that is invariant to specific aspects of the input data (Johansson et al., 2016; Kusner et al., 2017)

Original: [Table]
Revised: [Table]

Original: Existing work on counterfactual learning on text data expands upon this idea and aims at learning to discriminate causal signals from spurious correlations in learned representations of text data.
Revised: Existing work on counterfactual learning on text data expands upon this idea and aims at learning to discriminate causal signals from spurious correlations in learned representations of text data.

Original: Specifically, Choi et al.
Revised: Specifically, Choi et al.

Original: (2020b)
Revised: (2020b)

Original: ; Liang et al.
Revised: ; Liang et al.

Original: ; Choi et al.
Revised: ; Choi et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: (2022) show that using minimally different samples is helpful for learning robust representations that are invariant to spurious signals.
Revised: (2022) show that using minimally different samples is helpful for learning robust representations that are invariant to spurious signals.

Original: They generate counterfactual samples via dedicated masking strategies and apply contrastive learning on synthetic samples to discern counterfactual samples from original data.
Revised: They generate counterfactual samples via dedicated masking strategies and apply contrastive learning on synthetic samples to discern counterfactual samples from original data.

Original: However, few studies have delved into the effect of counterfactual learning on retrieval tasks (Choi et al., 2020a).
Revised: However, few studies have delved into the effect of counterfactual learning on retrieval tasks (Choi et al., 2020a).

Original: In this work, we reinterpret causal signals in existing work as answer spans in ODQA and seek to apply counterfactual contrastive learning for DPR.
Revised: In this work, we reinterpret causal signals in existing work as answer spans in ODQA and seek to apply counterfactual contrastive learning for DPR.

Original:  R ELEVANCE -A NSWERABILITY A SYNCHRONICITY ON DPR
Revised:  R ELEVANCE -A NSWERABILITY A SYNCHRONICITY ON DPR

Original: In this section, we first present the definition of counterfactual samples as unanswerable variants of the given passages.
Revised: In this section, we first present the definition of counterfactual samples as unanswerable variants of the given passages.

Original: We then conduct a counterfactual analysis on DPR where retrievers compare the relevance scores between an original passage and its counterfactual sample.
Revised: We then conduct a counterfactual analysis on DPR where retrievers compare the relevance scores between an original passage and its counterfactual sample.

Original: We show that dense retrievers are incapable of discriminating counterfactual samples from the original data and thus unaware of answerability.
Revised: We show that dense retrievers are incapable of discriminating counterfactual samples from the original data and thus unaware of answerability.

Original: 3. U NANSWERABLE C OUNTERFACTUAL S AMPLES FOR P ASSAGE R ETRIEVAL
Revised: 3. U NANSWERABLE C OUNTERFACTUAL S AMPLES FOR P ASSAGE R ETRIEVAL

Original: [Figure]
Revised: [Figure]

Original: Following previous studies (Liang et al., 2020; Choi et al., 2022), we synthesize counterfactual samples by removing causal signals from the original sample.
Revised: Following previous studies (Liang et al., 2020; Choi et al., 2022), we synthesize counterfactual samples by removing causal signals from the original sample.

Original: Based on our assumption that answer spans serve as causal signals in ODQA, we remove the answer span to a question in the original passage to create its counterfactual sample.
Revised: Based on our assumption that answer spans serve as causal signals in ODQA, we remove the answer span to a question in the original passage to create its counterfactual sample.

Original: Formally, we define counterfactual samples for passage retrieval as follows:
Revised: Formally, we define counterfactual samples for passage retrieval as follows:

Original: Definition 1.
Revised: Definition 1.

Original: Let ( q, p ) denote a question-answer passage pair, where p =
Revised: Let ( q, p ) denote a question-answer passage pair, where p =

Original: [ s l ; a ; s r ] contains an answer span a to the question q and non-answer spans s l and s r .
Revised: [ s l ; a ; s r ] contains an answer span a to the question q and non-answer spans s l and s r .

Original: We define a counterfactual sample p ∗ as an unanswerable variant of p with minimal changes such that p ∗ =
Revised: We define a counterfactual sample p ∗ as an unanswerable variant of p with minimal changes such that p ∗ =

Original: [ s l ; s r ] .
Revised: [ s l ; s r ] .

Original: Intuitively, the counterfactual sample is an unanswerable variant of the original passage that loses relevant features to the question ( i.e. , a ).
Revised: Intuitively, the counterfactual sample is an unanswerable variant of the original passage that loses relevant features to the question ( i.e. , a ).

Original: In practice, we set a to be the evidence sentence that contains the exact answer to the question.
Revised: In practice, we set a to be the evidence sentence that contains the exact answer to the question.

Original: Figureprovides the overview of our counterfactual sampling process.
Revised: Figureprovides the overview of our counterfactual sampling process.

Original: 3. M EASURING A NSWER - AWARENESS OF P ASSAGE R ETRIEVAL
Revised: M EASURING A NSWER - AWARENESS OF P ASSAGE R ETRIEVAL

Original: Our experiment aims to re-examine the belief that dense retrievers are answer-aware , or that the relevance score sim ( q, p ) between q and p is synchronous with the answerability of q from p .
Revised: Our experiment aims to re-examine the belief that dense retrievers are answer-aware , or that the relevance score sim ( q, p ) between q and p is synchronous with the answerability of q from p .

Original: Given that counterfactual samples lack key features relevant to the question, it is generally assumed that dense retrievers rank positive passages higher than their counterfactual counterparts.
Revised: Given that counterfactual samples lack key features relevant to the question, it is generally assumed that dense retrievers rank positive passages higher than their counterfactual counterparts.

Original: To assess the answer-awareness of dense retrieval approaches, we check whether such assumption holds true.
Revised: To assess the answer-awareness of dense retrieval approaches, we check whether such assumption holds true.

Original: Specifically, we observe answerability-relevance mismatch, where answerable passages p are given lower relevance scores than their counterfactual counterparts p ∗ .
Revised: Specifically, we observe answerability-relevance mismatch, where answerable passages p are given lower relevance scores than their counterfactual counterparts p ∗ .

Original: Definition 2.
Revised: Definition 2.

Original: (Answerability-Relevance Mismatch)
Revised: (Answerability-Relevance Mismatch)

Original: Given a question-passage pair ( q, p ) where p contains answer a to q , let p ∗ be the counterfactual sample of p such that a / ∈ p ∗ , and sim be the scoring function for relevance between q and p .
Revised: Given a question-passage pair ( q, p ) where p contains answer a to q , let p ∗ be the counterfactual sample of p such that a / ∈ p ∗ , and sim be the scoring function for relevance between q and p .

Original: Answerability-relevance mismatch between the question-passage pair ( q, p ) occurs if the relevance score sim ( q, p ) between q and p is lower than (or equal to)
Revised: Answerability-relevance mismatch between the question-passage pair ( q, p ) occurs if the relevance score sim ( q, p ) between q and p is lower than (or equal to)

Original: the score sim ( q, p ∗ ) between q and p ∗ , i.e. , sim ( q, p ) ≤ sim ( q, p ∗ )
Revised: the score sim ( q, p ∗ ) between q and p ∗ , i.e. , sim ( q, p ) ≤ sim ( q, p ∗ )

Original: Under the counterfactual simulation setting, we further introduce Answer-Awareness Rate of a model to quantify the rate at which the model predictions on relevance scores sim ( q, p ) are synchronous with the answerability of q to p .
Revised: Under the counterfactual simulation setting, we further introduce Answer-Awareness Rate of a model to quantify the rate at which the model predictions on relevance scores sim ( q, p ) are synchronous with the answerability of q to p .

Original: Given a set of T question-answer passage pairs { ( q i , p i ) } Ti =1 , answer-awareness rate counts the number of cases where answerability-relevance mismatch does not occur.
Revised: Given a set of T question-answer passage pairs { ( q i , p i ) } Ti =1 , answer-awareness rate counts the number of cases where answerability-relevance mismatch does not occur.

Original: Definition 3.
Revised: Definition 3.

Original: (Answer-Awareness Rate)
Revised: (Answer-Awareness Rate)

Original: Given a corpus C and a set of T triplets T = { ( q i , p i , p ∗ i )
Revised: Given a corpus C and a set of T triplets T = { ( q i , p i , p ∗ i )

Original: | p
Revised: | p

Original: i , p ∗ i ∈ C
Revised: i , p ∗ i ∈ C

Original: } Ti =1 where p ∗ i is the counterfactual sample of p
Revised: } Ti =1 where p ∗ i is the counterfactual sample of p

Original: i , let 1 sim ( q i ,p i ) ≤ sim ( q i ,p ∗ i ) be a binary indicator of whether answerability-relevance mismatch occurs for a triplet ( q i , p i , p ∗ i ) , i.e. , sim ( q i , p i ) ≤ sim ( q i , p ∗ i ) .
Revised: i , let 1 sim ( q i ,p i ) ≤ sim ( q i ,p ∗ i ) be a binary indicator of whether answerability-relevance mismatch occurs for a triplet ( q i , p i , p ∗ i ) , i.e. , sim ( q i , p i ) ≤ sim ( q i , p ∗ i ) .

Original: Answer-Awareness Rate, or AAR, is measured as the proportion of ( q i , p i , p ∗ i ) triplets whose relevance scores sim ( q i , p i ) between questions and original passages are higher than scores sim ( q i , p ∗ i ) between their counterfactual counterparts p ∗ .
Revised: Answer-Awareness Rate, or AAR, is measured as the proportion of ( q i , p i , p ∗ i ) triplets whose relevance scores sim ( q i , p i ) between questions and original passages are higher than scores sim ( q i , p ∗ i ) between their counterfactual counterparts p ∗ .

Original: [Equation]
Revised: [Equation]

Original: To validate the assumption on answer-awareness of dense retrievers, we observe how much AARs differ from their theoretical upper bound, which supposedly amounts to 100 percent provided that dense retrievers always measure relevance scores such that sim ( q, p ) >
Revised: To validate the assumption on answer-awareness of dense retrievers, we observe how much AARs differ from their theoretical upper bound, which supposedly amounts to 100 percent provided that dense retrievers always measure relevance scores such that sim ( q, p ) >

Original: sim ( q, p ∗ ) .
Revised: sim ( q, p ∗ ) .

Original: 3. P RELIMINARY E XPERIMENTS
Revised: 3. P RELIMINARY E XPERIMENTS

Original: We conduct our preliminary experiment on Natural Questions (NQ) (Kwiatkowski et al., 2019), a commonly used ODQA benchmark.
Revised: We conduct our preliminary experiment on Natural Questions (NQ) (Kwiatkowski et al., 2019), a commonly used ODQA benchmark.

Original: We follow the settings from Karpukhin et al.
Revised: We follow the settings from Karpukhin et al.

Original: and adopt
Revised: and adopt

Original: NQ test set used for DPR, which provides 3,610 factoid questions and around 21 million passages segmented from Wikipedia dump for evaluation.
Revised: NQ test set used for DPR, which provides 3,610 factoid questions and around 21 million passages segmented from Wikipedia dump for evaluation.

Original: To measure AAR, we use the 1,382 questions from
Revised: To measure AAR, we use the 1,382 questions from

Original: NQ test set whose gold ( q, p, p ∗ ) triplets are given.
Revised: NQ test set whose gold ( q, p, p ∗ ) triplets are given.

Original: 1 We then track whether the relevance score sim ( q, p ) between each question q and its positive passage p is higher than the score sim ( q, p ∗ ) between the counterfactual passage p ∗ .DPR
Revised: 1 We then track whether the relevance score sim ( q, p ) between each question q and its positive passage p is higher than the score sim ( q, p ∗ ) between the counterfactual passage p ∗ .

Original: Figure 2 shows both AARs and the retrieval performance of dense retrievers.
Revised: Figure 3.3 shows both AARs and the retrieval performance of dense retrievers.

Original: We first observe that AARs of  DPR models fall into the range of 68-75% and fail to reach the theoretical upper bound, which contradicts the assumption that retrievers constantly rank positive passages higher than counterfactual passages.
Revised: We first observe that AARs of a vanilla DPR significantly falls behind the theoretical upper bound, which contradicts the assumption that retrievers constantly rank positive passages higher than counterfactual passages.

Original: We also seea positive correlation between AAR and the retrieval performance of dense retrievers, suggesting that answer-unawareness of dense retrievers may be a bottleneck in  the effectiveness ofthe retriever-reader framework.
Revised: We also see that AAR of dense retrievers show positive correlations with both their retrieval performance and end-to-end QA accuracy. A considerable increase in QA performances following increase in AAR further suggests that answer-aware retrievers improve the effectiveness of retriever-reader frameworks even with a small gain in retrieval performance.

Original: Effect of Hard Negative Sampling.
Revised: Effect of Hard Negatives.

Original: One possible solution to improve AAR ofdense retrievers is to adopt better negative mining strategies.
Revised: One possible solution to improve AARs of retrievers is to adopt better negative mining strategies.

Original: To assess the effect of hard negative sampling, we compare AARs of DPR inbatch and DPR bm25 , which are trained with in-batch negatives and hard negatives from BM25 index, respectively.
Revised: To assess the effect of hard negatives, we compare AARs of DPR inbatch and DPR bm25 , which are trained with in-batch negatives and hard negatives from BM25 index, respectively.

Original: We also compare AARs of DPRmodels with those of ANCE and RocketQA, which adoptmore sophisticated negative sampling strategies to train a retriever.
Revised: We also compare AARs of DPR with those of ANCE and RocketQA, which adopt sophisticated negative sampling strategies for retriever training.

Original: From 
Revised: From Figure 3.3, we observe that

Original: Figure 2, we observe that DPR bm25 show better AAR than DPR inbatch , and that ANCE and RocketQA achieve better AAR than DPR.
Revised: DPR bm25 show better AAR than DPR inbatch , and that ANCE and RocketQA achieve better AAR than DPR.

Original: However, these approches require large computational cost since they depend on large PLMs and negative mining strategies which usually involve multi-stage training of different encoders.
Revised: However, these approaches require large computational cost since they depend on large PLMs and negative mining strategies which usually involve multi-stage training of different encoders.

Original: Effect of Sample Size.
Revised: Effect of Sample Size.

Original: Another possible solution is to increase the number of training samples.
Revised: Another possible solution is to increase the number of training samples.

Original: To examine the correlation between the size of the train set and AAR of the resulting DPR model, we randomly sample 15,000 and 30,000 question-passage pairs from NQ train set and additionally train two DPR models with randomly sampled data.
Revised: To examine the correlation between the size of the train set and AAR of the resulting DPR model, we randomly sample 15,000 and 30,000 question-passage pairs from NQ train set and additionally train two DPR models with randomly sampled data.

Original: Figure 2 shows that AAR of DPR increases when trained on more question-passage pairs.
Revised: Figure 3.3 shows that AAR of DPR increases when trained on more question-passage pairs.

Original: Despite such correlation, there is a practical limit to the size of the collected dataset, which makes any such approach infeasible.
Revised: Despite such correlation, there is a practical limit to the size of the collected dataset, which makes any such approach infeasible.

Original: Later in this paper we present our data-centric approach based on counterfactual augmentation,
Revised: Later in this paper we present our data-centric approach based on counterfactual augmentation,

Original: PiCL, which shows comparable AAR to ANCE and RocketQA in Figure 2. Note that, in this figure, DPR+PiCL adopts the same settings with DPR inbatch without relying on negative mining ( e.g. , hard negatives from BM25 or iterative training framework) or larger training set.
Revised: PiCL. Note that in Figure , a DPR+PiCL shows better AAR than a DPR trained under the same settings, i.e

Original:  S YNCHRONIZING R ELEVANCE AND A NSWERABILITY ON DPR
Revised:  S YNCHRONIZING R ELEVANCE AND A NSWERABILITY ON DPR

Original: Having seen that dense retrievers are not robust to unanswerable counterfactual passages, in this section, we present a novel contrastive learning approach that repurposes counterfactual samples to train a retriever robust to the counterfactual samples.
Revised: Having seen that dense retrievers are not robust to unanswerable counterfactual passages, in this section, we present a novel contrastive learning approach that repurposes counterfactual samples to train a retriever robust to the counterfactual samples.

Original: In Section 4.1, we redefine counterfactual samples as pivots between positive and negative passages to adapt them into a plain DPR setting.
Revised: In Section 4.1, we redefine counterfactual samples as pivots between positive and negative passages to adapt them into a plain DPR setting.

Original: In
Revised: In

Original: Section 4.2, we propose counterfactually- Pi voting C ontrastive L earning, which learns the relative similarity between questions, positives, negatives, and pivoting samples.
Revised: Section 4.2, we propose counterfactually- Pi voting C ontrastive L earning, which learns the relative similarity between questions, positives, negatives, and pivoting samples.

Original: 4. R EDEFINING C OUNTERFACTUAL S AMPLES AS P IVOTS
Revised: 4. R EDEFINING C OUNTERFACTUAL S AMPLES AS P IVOTS

Original: We have reinterpreted causal signals in dense retrieval as answer spans in ODQA and synthesized unanswerable counterfactual samples in Section 3.1.
Revised: We have reinterpreted causal signals in dense retrieval as answer spans in ODQA and synthesized unanswerable counterfactual samples in Section 3.1.

Original: Our goal is to incorporate counterfactual contrastive learning into DPR training such that the learned model is invariant to answer-irrelevant features.
Revised: Our goal is to incorporate counterfactual contrastive learning into DPR training such that the learned model is invariant to answer-irrelevant features.

Original: For that, we aim to design a method where the model learns the relative positions of counterfactual samples on the embedding space from DPR.
Revised: For that, we aim to design a method where the model learns the relative positions of counterfactual samples on the embedding space from DPR.

Original: Consider a question-passage pair ( q i , p + i ) , its corresponding counterfactual sample p ∗ i , and a set of
Revised: Consider a question-passage pair ( q i , p + i ) , its corresponding counterfactual sample p ∗ i , and a set of

Original: N negative passages { p − j } Nj =1 .
Revised: N negative passages { p − j } Nj =1 .

Original: A DPR-based retriever learns question and passage representations such that the following inequality between positive passages p + i and negative passages p − j holds:
Revised: A DPR-based retriever learns question and passage representations such that the following inequality between positive passages p + i and negative passages p − j holds:

Original: [Equation]
Revised: [Equation]

Original: To specify the relative positions of counterfactual passage on the embedding space, one must take into account their relevance between positive passages ( i.e. source passages) and negative passages.
Revised: To specify the relative positions of counterfactual passage on the embedding space, one must take into account their relevance between positive passages ( i.e. source passages) and negative passages.

Original: As shown in Figure 3, we view our counterfactual samples from two different perspectives, as hard negatives and as pseudo-positives.
Revised: As shown in Figure 3, we view our counterfactual samples from two different perspectives, as hard negatives and as pseudo-positives.

Original: Counterfactuals as Hard Negatives.
Revised: Counterfactuals as Hard Negatives.

Original: Given that counterfactual samples lack relevant features to the question, p ∗ i is a counterfactually negative sample to an anchor question q i while the original passage p + i serves as the positive.
Revised: Given that counterfactual samples lack relevant features to the question, p ∗ i is a counterfactually negative sample to an anchor question q i while the original passage p + i serves as the positive.

Original: Thus the embedding similarity sim ( q i , p ∗ i ) between q i and p ∗ i is upper bounded by sim [Equation]
Revised: Thus the embedding similarity sim ( q i , p ∗ i ) between q i and p ∗ i is upper bounded by sim ( q i , p + i ) :

Original: [Equation]
Revised: [Equation]

Original: Learning to discriminate p ∗ i from p
Revised: Learning to discriminate p ∗ i from p

Original: + i would minimize the mutual information between representations of questions
Revised: + i would minimize the mutual information between representations of questions

Original: q i and answer-irrelevant spans in p + i .
Revised: q i and answer-irrelevant spans in p + i .

Original: Counterfactuals as Pseudo Positives.
Revised: Counterfactuals as Pseudo Positives.

Original: Since a counterfactual passage p ∗ i is a minimally different sample that retains most of semantics in the positive passage p
Revised: Since a counterfactual passage p ∗ i is a minimally different sample that retains most of semantics in the positive passage p

Original: + i , p ∗ i can be seen as a pseudo-positive example in the passage retrieval.
Revised: + i , p ∗ i can be seen as a pseudo-positive example in the passage retrieval.

Original: Semantic relevance between q i and p ∗ i distinguishes p ∗ i from other negatives p − j , which provide noisy contexts with respect to q i .
Revised: Semantic relevance between q i and p ∗ i distinguishes p ∗ i from other negatives p − j , which provide noisy contexts with respect to q i .

Original: Thus, the following holds for all p − j :
Revised: Thus, the following holds for all p − j :

Original: [Equation]
Revised: [Equation]

Original: Learning to discriminate p ∗ i from p − j imposes a constraint on the encoders such that embeddings for questions and passages are computed based on their semantic alignment.
Revised: Learning to discriminate p ∗ i from p − j imposes a constraint on the encoders such that embeddings for questions and passages are computed based on their semantic alignment.

Original: From Equation 5 and 6, we can derive that the embedding similarity sim ( q i , p ∗ i ) between questions and counterfactual samples are bounded by sim ( q i , p + i ) and sim ( q i , p − j ) .
Revised: From Equation 5 and 6, we can derive that the embedding similarity sim ( q i , p ∗ i ) between questions and counterfactual samples are bounded by sim ( q i , p + i ) and sim ( q i , p − j ) .

Original: Essentially, they can be re-formulated as pivots between positive and negative samples in the embedding space.
Revised: Essentially, they can be re-formulated as pivots between positive and negative samples in the embedding space.

Original: Since both inequality constraints in Equation 5 and 6 satisfy Equation 4, our definition of counterfactual samples as pivots is in line with the training objective of DPR.
Revised: Since both inequality constraints in Equation 5 and 6 satisfy Equation 4, our definition of counterfactual samples as pivots is in line with the training objective of DPR.

Original: 4. C OUNTERFACTUALLY -P IVOTING C ONTRASTIVE L EARNING
Revised: 4. C OUNTERFACTUALLY -P IVOTING C ONTRASTIVE L EARNING

Original: Based on the ideas discussed in Section 4.1, we propose counterfactually-
Revised: Based on the ideas discussed in Section 4.1, we propose counterfactually-

Original: Pi voting C ontrastive
Revised: Pi voting C ontrastive

Original: L earning ( PiCL ) for dense retrieval for a robust, answer-aware dense retriever.
Revised: L earning ( PiCL ) for dense retrieval for a robust, answer-aware dense retriever.

Original: Specifically, we introduce additional counterfactual contrastive loss terms into the passage retrieval objective function in Karpukhin et al.
Revised: Specifically, we introduce additional counterfactual contrastive loss terms into the passage retrieval objective function in Karpukhin et al.

Original: (2020) to leverage counterfactual samples as pivots between positives and negatives.
Revised: (2020) to leverage counterfactual samples as pivots between positives and negatives.

Original: Consider a question-positive passage pair { q i , p + i } , a set of N negatives { p − j } Nj ̸
Revised: Consider a question-positive passage pair { q i , p + i } , a set of N negatives { p − j } Nj ̸

Original: = i , and their corresponding counterfactual samples p ∗ i and { p ∗ j} N
Revised: = i , and their corresponding counterfactual samples p ∗ i and { p ∗ j

Original: j ̸
Revised: } Nj ̸

Original: = i .
Revised: = i .

Original: We define following loss terms as key components of PiCL:
Revised: We define following loss terms as key components of PiCL:

Original: Modified Dense Passage Retrieval Loss.
Revised: Modified Dense Passage Retrieval Loss.

Original: The modified DPR loss L dpr is a slight modification of the loss term in Karpukhin et al.
Revised: The modified DPR loss L dpr is a slight modification of the loss term in Karpukhin et al.

Original: (2020) in which the counterfactual counterpart p ∗ i to the positive passage p
Revised: (2020) in which the counterfactual counterpart p ∗ i to the positive passage p

Original: + i is added as a negative.
Revised: + i is added as a negative.

Original: To alleviate any interference from masked passages as negatives, we add a balancing coefficient λ < 1 before the similarity term e sim ( q i ,p ∗ i ) .
Revised: To alleviate any interference from masked passages as negatives, we add a balancing coefficient λ < 1 before the similarity term e sim ( q i ,p ∗ i ) .

Original: [Equation]
Revised: [Equation]

Original: Counterfactuals as Hard Negatives.
Revised: Counterfactuals as Hard Negatives.

Original: The Counterfactuals-as-Hard-Negatives loss L chn is optimized to maximize the similarity between q i and p
Revised: The Counterfactuals-as-Hard-Negatives loss L chn is optimized to maximize the similarity between q i and p

Original: + i while minimizing the similarity between q i and p ∗ i .
Revised: + i while minimizing the similarity between q i and p ∗ i .

Original: It imposes the key constraint on q i , p + i , and p ∗ i from Equation 5 to discriminate answer passages from non-answer counterfactual passages:
Revised: It imposes the key constraint on q i , p + i , and p ∗ i from Equation 5 to discriminate answer passages from non-answer counterfactual passages:

Original: [Equation]
Revised: [Equation]

Original: Counterfactuals as Pseudo Positives.
Revised: Counterfactuals as Pseudo Positives.

Original: The Counterfactuals-as-Pseudo-Positives loss L cpp is optimized to maximize the relative similarity between q i and p ∗ i with respect to negative passages p − j and p ∗ j in the given batch.
Revised: The Counterfactuals-as-Pseudo-Positives loss L cpp is optimized to maximize the relative similarity between q i and p ∗ i with respect to negative passages p − j and p ∗ j in the given batch.

Original: The key difference between L dpr and L cpp is that the counterfactual passage is used as a positive in L cpp to retain semantic relevance in the learned embeddings.
Revised: The key difference between L dpr and L cpp is that the counterfactual passage is used as a positive in L cpp to retain semantic relevance in the learned embeddings.

Original: [Equation]
Revised: [Equation]

Original: The final loss function L is a weighted sum of all three loss fuctions L dpr , L chn , and L cpp :
Revised: The final loss function L is a weighted sum of all three loss fuctions L dpr , L chn , and L cpp :

Original: [Equation]
Revised: [Equation]

Original: where τ 1 , τ 2 are hyperparameters that determine the importance of the terms.
Revised: where τ 1 , τ 2 are hyperparameters that determine the importance of the terms.

Original: Later in Section 5.
Revised: Later in Section 5.

Original: we provide an analysis on the effect of each loss term through an ablation study.
Revised: we provide an analysis on the effect of each loss term through an ablation study.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first evaluate the effectiveness of PiCL for passage retrieval task on common
Revised: In this section, we first evaluate the effectiveness of PiCL for passage retrieval task on common

Original: ODQA benchmarks.
Revised: ODQA benchmarks.

Original: We then apply PiCL on a retriever-reader pipeline and assess its end-to-end
Revised: We then apply PiCL on a retriever-reader pipeline and assess its end-to-end

Original: QA performance to show that improving answer-awareness of the retriever leads to a more accurate reader.
Revised: QA performance to show that improving answer-awareness of the retriever leads to a more accurate reader.

Original: We also provide some analysis on PiCL as possible explanations to our design choice.
Revised: We also provide some analysis on PiCL as possible explanations to our design choice.

Original: 5. E XPERIMENTAL S ETTINGS 
Revised: 5. E XPERIMENTAL S ETTINGS Dataset.

Original: Dataset.

Original: Following the settings in Sections 3.3, we evaluate our method on Natural Questions (kwi).
Revised: Following the settings in Sections 3.3, we evaluate our method on Natural Questions (Kwiatkowski et al., 2019).

Original: This benchmark is built upon web documents such as Wikipedia articles and human-annotated question-answer pairs, some of which are collected from real-world search queries (kwi).
Revised: This benchmark is built upon web documents such as Wikipedia articles and human-annotated question-answer pairs, most of which are collected from real-world search queries (Kwiatkowski et al., 2019).

Original: We follow prior work on dense retrieval (Xiong et al., 2021;
Revised: We follow prior work on dense retrieval (Xiong et al., 2021;

Original: Qu et al., 2021;
Revised: Qu et al., 2021;

Original: Ren et al., 2021) and use for training 58,812 factoid questions each paired with a set of positive passages.
Revised: Ren et al., 2021) and use for training 58,812 factoid questions each paired with a set of positive passages.

Original: For inference, we use the preprocessed data from Karpukhin et al.
Revised: For inference, we use the preprocessed data from Karpukhin et al.

Original: (2020), which include 3,610 factoid questions and a corpus of about 21M Wikipedia passages preprocessed through the pipeline proposed in Chen et al.
Revised: (2020), which include 3,610 factoid questions and a corpus of about 21M Wikipedia passages preprocessed through the pipeline proposed in Chen et al.

Original: Baselines and Implementation Details.
Revised: Baselines and Implementation Details.

Original: We consider DPR (Karpukhin et al., 2020) as the backbone architecture for all models implemented in this section.
Revised: We consider DPR (Karpukhin et al., 2020) as the backbone architecture for all models implemented in this section.

Original: Our focus is to assess how applying PiCL affects the downstream performance of the backbone DPR.
Revised: Our focus is to assess how applying PiCL affects the downstream performance of the backbone DPR.

Original: We also include ANCE (Xiong et al., 2021) and RocketQA (Qu et al., 2021), which use dedicated negative sampling pipelines to improve the performance of vanilla DPR.
Revised: We also include ANCE (Xiong et al., 2021) and RocketQA (Qu et al., 2021), which use dedicated negative sampling pipelines to improve the performance of vanilla DPR.

Original: To show the orthogonality of PiCL to such negative mining approaches, we implement PiCL models under different negative sampling strategies ( e.g. , sampling from top retrieval results of BM25 and a fine-tuned DPR) and compare them with the baselines.
Revised: To show the orthogonality of PiCL to such negative mining approaches, we implement PiCL models under different negative sampling strategies ( e.g. , sampling from top retrieval results of BM25 and a fine-tuned DPR) and compare them with the baselines.

Original: Note that due to limited computation resource, we do not reproduce the performance of ANCE and
Revised: Note that due to limited computation resource, we do not reproduce the performance of ANCE and

Original: RocketQA.
Revised: RocketQA.

Original: See Appendix A for more details on the implementation.
Revised: See Appendix A for more details on the implementation.

Original: 5. M AIN E XPERIMENTAL R ESULTS 
Revised: 5. M AIN E XPERIMENTAL R ESULTS Performance of Passage Retrievers.

Original: Performance of Passage Retrievers.

Original: Table 2 compares the performance of PiCL models with the baselines, i.e. , DPR, ANCE, and RocketQA on NQ benchmark.
Revised: Table 2 compares the performance of PiCL models with the baselines, i.e. , DPR, ANCE, and RocketQA on NQ benchmark.

Original: We observe that DPR+PiCL trained using in-batch negatives achieves performance comparable to or even better than all baseline DPR † including those trained with additional hard negatives.
Revised: We observe that DPR+PiCL models yield consistent performance gains over vanilla DPR under all tested conditions.

Original: Similar to DPR, DPR+PiCL shows particularly strong performance when trained with hard negatives from top retrieval results of a fine-tuned DPR, which suggests that the performance of DPR+PiCL can be further boosted by adding informative negative samples.
Revised: Similar to DPR, DPR+PiCL shows particularly strong performance when trained with hard negatives from top retrieval results of a fine-tuned DPR, which suggests that the performance of DPR+PiCL can be further boosted by adding informative negative samples.

Original: 2 Overall, we see that DPR+PiCL trainedwith hard negative mined usingDPR index achieves performance second only to model-centric approaches, i.e. , ANCE and RocketQA, which rely on larger pretrained language models ( e.g. , ERNIE  and RoBERTa) and compute-intensive negative mining techniques.
Revised: Meanwhile, we see a consistent increase in the performance of dense retrievers when trained using model-centric approaches, i.e. , ANCE and RocketQA, which rely on larger pretrained language models ( e.g. , ERNIE (Sun et al., 2020) and RoBERTa (Zhuang et al., 2021)) and compute-intensive negative mining techniques.

Original: The performance gain on DPR+PiCL when using additional hard negativesfurther implies that PiCL can be  orthogonally applied to such model-centric approaches. 
Revised: The performance gain on DPR+PiCL when using additional hard negatives implies that PiCL can be further improved when orthogonally applied to such model-centric approaches. -hn-train , which is available on

Original: [Table]
Revised: [Table]

Revised: train retrieved passages inference

Revised: Note that we reuse the DPR

Revised: Fusion

Original: [Table]
Revised: [Table]

Original: End-to-End QA Performance of Retriever-Reader.
Revised: End-to-End QA Performance of Retriever-Reader.

Original: A key assumption underlying our work is that synchronizing a retriever with the reader improves end-to-end QA performance of the retrieverreader pipeline.
Revised: A key assumption underlying our work is that synchronizing a retriever with the reader improves end-to-end QA performance of the retrieverreader pipeline.

Original: To validate this assumption, we incorporate DPR+PiCL into an extractive QA system and evaluate the downstream performance of the subsequent reader.
Revised: To validate this assumption [Table] we incorporate DPR+PiCL into a QA system and evaluate the downstream performance of the subsequent reader.

Original: Specifically, we re-use theextractive reader  from Karpukhin et al.
Revised: Specifically, we re-use the reader models from Karpukhin et al.

Revised: (2020) and Izacard & Grave (2021b)

Original: and switch different retrievers to sample top-
Revised: and switch different retrievers to sample top-

Original: k passages for reader inference.
Revised: k passages for reader inference.

Original: We then compute Exact Match (EM) scores for the reader, which measures the proportion of questions whose answer prediction is equivalent to correct answers.
Revised: We then compute Exact Match (EM) scores for the reader, which measures the proportion of questions whose answer prediction is equivalent to correct answers.

Original: Table 3 reports end-to-end QA performance of the retriever-reader pipelines.
Revised: Table 3 reports end-to-end QA performance of the retriever-reader pipelines.

Revised: Overall, applying PiCL on DPR consistently improves the QA performance of the retriever-reader pipeline under various settings.

Revised: Particularly, we see that using a FiD base instead of a DPR reader for

Original: Applying PiCL on DPR significantly improves both retrieval and QA performance of the retriever-reader pipeline, particularly when k is small.
Revised: DPR+PiCL further boosts the performance of the QA system, suggesting that PiCL models can be orthogonally applied with advanced reader models. [Figure] [Table] [Figure]

Original: We find that such performance gap between DPR and DPR+PiCL is consistent across all settings on negative samples, which implies that the performance gain from PiCL is independent of negative sampling strategies.
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: 5. A NALYSIS ON D ENSE P ASSAGE R ETRIEVAL
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: 5. A NALYSIS ON D ENSE P ASSAGE R ETRIEVAL

Original: In-batch Negatives.
Revised: Negative Samples.

Original: Figure 4 shows the retrieval accuracy of DPR and DPR+PiCL trained with different amounts of in-batch  negative samples.
Revised: Figure 4 shows the retrieval accuracy of DPR and DPR+PiCL trained with different amounts of in-batch and random negative samples 3 .

Original: We see a consistent increase in retrieval accuracy for DPR+PiCL, which is in line with the finding that the performance gain from PiCL  is independent of the numberand quality of negative samples used to train the model.
Revised: We see a consistent increase in retrieval accuracy for DPR+PiCL, which is in line with the finding that the performance gain from PiCL over a vanilla DPR is independent of the number of negative samples used to train the model.

Original: Particularly, DPR+PiCL trained with less in-batch negatives achieves performance comparable to or better than a vanilla DPR trained with morein-batch negatives, as shown in Figure 4.
Revised: Particularly, DPR+PiCL trained with fewer negatives achieves performance comparable to or better than a vanilla DPR trained with more negatives, as shown in Figure 4.

Original: One possible reason for such efficiency is thatPiCL using counterfactual samples  leads to the effect of data augmentation as learning to discriminate positive passages from hard negative passages.
Revised: One possible reason for such efficiency is that using counterfactual samples for PiCL leads to the effect of data augmentation as the model learns to discriminate positive passages from hard negative passages.

Original: Counterfactual Loss.
Revised: Counterfactual Loss.

Original: To study the efficacy of counterfactual samples as pivots, we conduct an ablation study on counterfactual contrastive learning objective in PiCL (Equation 10).
Revised: To study the efficacy of counterfactual samples as pivots, we conduct an ablation study on counterfactual contrastive learning objective in PiCL (Equation 10).

Original: Specifically, we implement three PiCL baselines with the following modifications on the objective function in
Revised: Specifically, we implement three PiCL baselines with the following modifications on the objective function in

Original: Equation 10, 1) L dpr + L cpp , 2)
Revised: Equation 10, 1) L dpr + L cpp , 2)

Original: L dpr + L chn , and 3) L cpp + L chn .
Revised: L dpr + L chn , and 3) L cpp + L chn .

Original: Table 4 compares all baselines with the original PiCL and the vanilla DPR.
Revised: Table 4 compares all baselines with the original PiCL and the vanilla DPR.

Original: We find that all three PiCL baselines do not improve much over a vanilla DPR without either L dpr , L cpp or L chn , while the original PiCL method significantly outperforms the vanilla DPR.
Revised: All three PiCL baselines do not improve much over a vanilla DPR without either L dpr , L cpp or L chn .

Original: This suggests that using counterfactual samples as pivots is crucial in applying counterfactual contrastive learning to DPR training.
Revised: In contrast, the original PiCL method outperforms the vanilla DPR, suggesting that using counterfactual samples as pivots is crucial in PiCL training.

Original: Counterfactual Sampling Strategy.
Revised: Counterfactual Sampling Strategy.

Original: To validate our strategy of synthesizing counterfactual samples, we present different augmentation schemes and assess their effect on downstream performance of DPR+PiCL.
Revised: To validate our strategy of synthesizing counterfactual samples, we present different augmentation schemes and assess their effect on downstream performance of DPR+PiCL.

Original: Specifically, we consider the following four augmentation schemes: 1) removing the exact answer match, PiCL answer , 2) removing an extended span around the answer, PiCL window , and 3) removing the answer sentence, PiCL sentence .
Revised: Specifically, we consider the following four augmentation schemes: 1) removing the exact answer match, PiCL answer , 2) removing an extended span around the answer, PiCL window , and 3) removing the answer sentence, PiCL sentence .

Original: Table 4 reports the retrieval performance of
Revised: Table 4 reports the retrieval performance of

Original: PiCL models trained under different masking strategies.
Revised: PiCL models trained under different masking strategies.

Original: Overall, PiCL sentence yields the best performance among all models.
Revised: Overall, PiCL sentence yields the best performance among all models.

Original: This implies that not only answer string but also its context is an important factor in learning answer-awareness.
Revised: This implies that not only answer string but also its context is an important factor in learning answer-awareness.

Original: Answer-awareness.
Revised: Answer-awareness.

Original: One key assumption underlying our approach is that applying PiCL on DPR results in a more answer-aware retriever.
Revised: One key assumption underlying our approach is that applying PiCL on DPR results in a more answer-aware retriever.

Original: To provide a fine-grained analysis on answer-awareness of PiCL, we use DPR+PiCL trained with in-batch negatives to measure similarity scores sim ( q, p + ) and sim ( q, p ∗ ) for the gold ( q, p + , p ∗ ) triplets from Section 3.3.
Revised: To provide a fine-grained analysis on answer-awareness of PiCL, we use DPR+PiCL trained with inbatch negatives to measure similarity scores sim ( q, p + ) and sim ( q, p ∗ ) for the gold ( q, p + , p ∗ ) triplets from Section 3.3.

Original: Figure 5 shows the average of the scores sim ( q, p + ) and sim ( q, p ∗ ) from DPR+PiCL compared to those from DPR.
Revised: Figure 5 shows the average of the scores sim ( q, p + ) and sim ( q, p ∗ ) from DPR+PiCL compared to those from DPR.

Original: We observe a notable difference between the average sim ( q, p + ) and sim ( q, p ∗ ) for DPR+PiCL and a relatively minor difference for DPR.
Revised: We observe a notable difference between the average sim ( q, p + ) and sim ( q, p ∗ ) for DPR+PiCL and a relatively minor difference for DPR.

Original: In DPR+PiCL, the average similarity score sim ( q, p + ) is significantly larger than the score sim ( q, p ∗ ) , which indicates that DPR+PiCL learns answer-awareness to distinguish between the answerable passage and the unanswerable counterfactual one.
Revised: In DPR+PiCL, the average similarity score sim ( q, p + ) is significantly larger than the score sim ( q, p ∗ ) , which indicates that DPR+PiCL learns answer-awareness to distinguish between the answerable passage and the unanswerable counterfactual one.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This work aims to re-examine the assumption that dense retrievers assign high relevance scores on answerable passages to questions.
Revised: This work aims to re-examine the assumption that dense retrievers assign high relevance scores on answerable passages to questions.

Original: We first conduct a counterfactual simulation and observe that dense retrievers based on DPR often fail to rank answerable passages higher than their counterfactual counterparts.
Revised: We first conduct a counterfactual simulation and observe that dense retrievers based on DPR often fail to rank answerable passages higher than their counterfactual counterparts.

Original: Based on this observation, we present PiCL, which repurposes counterfactual samples as pivots between positives and negatives for answer-aware DPR.
Revised: Based on this observation, we present PiCL, which repurposes counterfactual samples as pivots between positives and negatives for answer-aware DPR.

Original: Our experiments show that applying PiCL on DPR not only enhances the model performance on the ODQA benchmark but also improves its robustness to unanswerable, counterfactual passages.
Revised: Our experiments show that applying PiCL on DPR not only enhances the model performance on the ODQA benchmark but also improves its robustness to unanswerable, counterfactual passages.

Original: We believe that the concept of counterfactually-pivoting samples can be further explored in future work: 1) extending counterfactuals to multiple pivots of incrementally removing causal signals at different levels, 2) refining the definition of answer-awareness or AAR into formal evaluation metrics of ODQA, and 3) shifting a target task to other representation learning tasks such as text/image classification and response retrieval for dialogue systems.
Revised: We believe that the concept of counterfactually-pivoting samples can be further explored in future work: 1) extending counterfactuals to multiple pivots of incrementally removing causal signals at different levels, 2) refining the definition of answer-awareness or AAR into formal evaluation metrics of ODQA, and 3) shifting a target task to other representation learning tasks such as text/image classification and response retrieval for dialogue systems.

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: We train all implemented models for 25 epochs on a single server with two 16-core Intel(R)
Revised: In Table 2, we train all implemented models for 40 epochs on a single server with two 16-core Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6226R CPUs, a 264GB RAM, and 3 24GB GPUs.
Revised: Gold 6226R CPUs, a 264GB RAM, and 8 24GB GPUs.

Original: For PiCL training, we set batch size as 16, learning rate as 2e-5, and eps and betas of the adam optimizer as 1e-8 and (0.9, 0.999), respectively.
Revised: For PiCL training, we set batch size as 16, learning rate as 2e-5, and eps and betas of the adam optimizer as 1e-8 and (0.9, 0.999), respectively.

Revised: Note that we conduct experiments on the NQ benchmark under the same settings used in Karpukhin et al.

Original: Among the hyperparameters { 0 . 1 , 0 .
Revised: Among the hyperparameters { 0 . 1 , 0 .

Revised: , we choose 0.2 for the balancing coefficient λ for counterfactual samples in Equation 7 and 1 . 0 for the weight hyperparameters τ 1 , τ 2 in Equation 10.

Revised: For additional analysis in Section 5.3, we move onto a more viable settings with limited computation resources.

Revised: Specifically, we train all models for 25 epochs on a server with 3 24GB GPUs.

Original: 5 , 0 . 9 } , we choose 0.2 for the balancing coefficient λ for counterfactual samples in Equation 7 and 0.2 for the weight hyperparameters τ 1 , τ 2 in Equation 10. 
Revised: We choose 0.2 for the balancing coefficient λ for counterfactual samples in Equation 7 and 0.2 for the weight hyperparameters τ 1 , τ 2 in Equation 10. All other hyperparameters including batch size and optimizer remain unchanged.

Original: For reader in Section 5.2, we adopt the extractive reader from Karpukhin et al.
Revised: For reader in Section 5.2, we consider two models: 1) the extractive reader from Karpukhin et al.

Original: (2020), implemented on pretrained BERT models (Devlin et al., 2019) and coupled with DPR models. 
Revised: (2020) implemented on pretrained BERT models (Devlin et al., 2019) and 2) Fusion-in-Decoder reader (Izacard & Grave, 2021b) based on pretrained T5-base (Raffel et al., 2020) models. We conduct inference for the reader on a single 24GB GPU with the batch size of 8. B A DDITIONAL A NALYSIS OF A NSWER - AWARENESS

Original: All reader models are trained for 18 epochs with batch size 8.
Revised: In this section, we extend our preliminary experiment in Section 3.2 and delve into two important aspects of answer-aware retrievers: 1) Score Differences and 2) Effect of Question Types. [Table] Score To provide an overview of how relevance scores differ between original and counterfactual passages, we expand upon the answer-awareness analysis in Section 5.3 and focus on the difference in relevance scores sim ( q, p + ) and sim ( q, p ∗ ) .

Revised: Specifically, we aim to examine whether DPR+PiCL tends to put larger differences between original and counterfactual passages compared to DPR.

Revised: For that, we reuse the gold ( q, p + , p ∗ ) triplets from Section 3.3 and compute the score difference sim ( q, p + ) − sim ( q, p ∗ ) for each ( q, p + , p ∗ ) .

Revised: The relevance scores are measured using both

Revised: DPR and DPR+PiCL for a comparative analysis.

Revised: Table 5 compares the average score differences measured from DPR and DPR+PiCL.

Revised: For our analysis, we consider two subsets of samples, one that satisfies sim ( q, p + ) >

Revised: sim ( q, p ∗ ) ( i e. Answer-aware) and another that doesn’t ( i e. Answer-unaware).

Revised: We observe that training DPR with PiCL generally leads to larger score differences than a DPR, particularly for samples where the retrievers are answer-aware, i.e. sim ( q, p + ) >

Revised: sim ( q, p ∗ ) .

Revised: In line with the previous observations from the AAR experiment in Section 3.3, the results in Table 5 suggest that DPR+PiCL learns to distinguish answerable passages from counterfactual samples.

Revised: While the differences may seem small, we note that a slight decrease in the relevance score leads to a large drop in the rank of a passage.

Revised: We further analyze the distributions of score differences and examine how such distributions affect the answer-awareness of retrievers.

Revised: For that, we plot and compare the distribution of score differences from DPR and DPR+PiCL.

Revised: We see from Figure 6 that the distribution of sim ( q, p + ) − sim ( q, p ∗ ) drawn from DPR+PiCL is skewed more positive than negative, while the distribution from DPR tends to be more symmetrical.

Revised: Negative skewness of the distribution from DPR+PiCL indicates that 1) more ( q, p + , p ∗ ) samples satisfy sim ( q, p + )

Revised: > sim ( q, p ∗ ) and that 2) the retriever tends to assign higher relevance scores on p + .

Revised: This result in turn suggests that DPR+PiCL is more answer-aware, i.e. , capable of distinguishing answerable passages from counterfactuals.

Revised: Effect of Question Types.

Revised: To better understand if the answer-awareness of a retriever depends on certain linguistic features, we start by investigating the effect of question types on the answerawareness of DPR.

Revised: Specifically, we reuse the gold ( q, p + , p ∗ ) triplets from AAR experiments and classify them into six subsets based on question types.

Revised: We identify the type of a question simply by looking for exact matches of the question words: how, what, when, where, which, and who.

Revised: [Table]

Revised: When

Revised: [Table]

Revised: Where

Revised: [Table]

Revised: Which

Revised: [Table]

Revised: Who

Revised: [Table]

Revised: Table 6 shows AARs of DPR and DPR+PiCL on the six subsets of different question types.

Revised: We first observe that AARs of dense retrievers do vary significantly across different question types, implying that the answer-awareness of DPR depends on the types of questions being asked.

Revised: Particularly, DPR shows large gaps in AARs between different question types, ranging from 65.07% to 80.68%.

Revised: On the other hand, DPR+PiCL shows 1) significant improvements in AAR over a vanilla DPR for all question types and 2) smaller variance in AARs across different question types.

Revised: Question: Whodiedintheplanecrashgreysanatomy

Revised: PassageType

Original: We conduct inference for the reader on 2 24GB GPUs with the batch size of 10.
Revised: Title Text [Table] Among all question types, we see that DPR shows particularly low AARs on “where”, “what”, and “who” questions.

Revised: These questions tend to include answers that refer to named entities, i.e. names of people, locations, and objects.

Revised: Our hypothesis is that DPR fails to identify the presence of target entities, which serve as causal features and clues to answer passages.

Revised: Table 7 shows an example from the retrieval result of illustrates the problem of named entities for DPR.

Revised: While DPR is capable of retrieving passages with relevant semantics such as aircraft crash, plane, and killing , it fails to identify key named entities in the question such as Greys Anatomy .

Revised: Contrary to a vanilla DPR,

Revised: DPR+PiCL learns to differentiate answer passages from their counterfactual counterparts in which key entities are not present.

Revised: In fact, we see from the example in Table 7 that DPR+PiCL is more inclined to identify answer-relevant features, i.e. , named entities.

Revised: In some sense, our approach is in line with previous methods based on salient span masking (Guu et al., 2020) such as MSS (Sachan et al., 2021), where the retriever is trained by predicting masked salient spans like named entities with the help of a reader.

Revised: C C HOICE OF P RETRAINED L ANGUAGE M ODELS

Revised: Previous studies on dense retrieval, including RocketQA (Qu et al., 2021), show that using a better PLM such as ERNIE (Sun et al., 2020) for retriever training leads to better performance.

Revised: To study the effect of the choice of PLMs on the performance of retrievers trained with PiCL, we have implemented DPR+PiCL models with pretrained ERNIE and measured their performance on the

Revised: NaturalQuestions benchmark.

Revised: [Table]

Revised: Based on table 8, we observe that the ERNIE implementation of DPR+PiCL outperforms its original BERT implementation, and that its performance increases with the number of negatives (i.e. in-batch and BM25 negative passages) used to train the model.

Revised: While RocketQA does achieve a substantial performance gain over other baselines, it relies heavily on significantly large batches and a compute-intensive negative sampling pipeline, which makes it infeasible to reproduce the model due to tremendous computational costs.