Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions (2023)

Abstract: Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR’s contrast consistency is improved without sacrificing its accuracy on the standard test sets.1

## introduction contrast consistency @xcite is a crucial aspect for neural models in nlp. models are expected to identify perturbations in the text input and decide whether such a semantic shift leads to a different label. to evaluate this consistency, contrast sets have been introduced in various tasks such as sentiment analysis @xcite , natural language inference @xcite , and reading comprehension @xcite by minimally modifying the original input to reverse the original label. however, to our best knowledge, there is no study on the contrast consistency in open-domain question answering (openqa). in openqa, even a slight modification of a word or two can alter the meaning of the question, which leads to a completely different answer. to maintain contrast consistency, models are expected to predict the corresponding answer when such semantic shift occurs. studying contrast consistency in openqa poses unique challenges. firstly, collecting appropriate contrast sets is difficult. while contrast sets have been developed for reading comprehension @xcite , they typically replaced an entity (e.g., barack obama was born in hawaii) in given context with another entity (e.g., barack obama was born in new york), leading to a different answer to the given question (e.g., where was barack obama born?). constructing such contrast sets does not necessitate the factuality of the perturbed context, as the answer depends solely on the context rather than world knowledge. however, in the absence of evidence context, the perturbed questions in openqa must be factually answerable in accordance with world knowledge, which is beyond what rule-based methods can do. secondly, achieving contrast consistency is challenging for openqa models, which usually follow the ''retrieve-then-read'' pipeline @xcite . in addition to the challenge of predicting answers from a contrast context as in reading comprehension, models also face the challenge of mapping the perturbed question with its corresponding evidence passage in a large corpus. the latter requires the retriever to distinguish the minimal semantic difference between embeddings of the perturbed question and the original question, which is ignored in typical retriever training. to fill this gap in openqa, we propose to create contrast sets using minimally edited questions (meqs). given a question q and its answer a, an meq q is defined as a question that possesses high lexical and semantic similarity with q, while having a distinct answer a (@xmath0. for figure 1 : above: trained on question q 1 but not a contrast one q 2 , dpr generated an overly similar embedding of q 2 with q 1 's and thus falsely retrieved p 1 . we aim to identify q 2 as a distinct question and retrieve p 2 instead. below: the performance of dprbased openqa models on the standard nq question set and our contrast set of minimally edited questions (meqs). we find that the state-of-the-art openqa models which employ the dense passage retriever (dpr) @xcite struggle on our meq contrast sets. as shown in figure 1 , dpr-retrieved passages lead to 63% downstream qa accuracy on the training set and 43% on standard test set. however, the accuracy drops to 20%∼25% on our meq contrast sets. the problem lies in the contrastive training process of dpr. the model is trained to optimize question embeddings to be closer to their positive passage embeddings foot_1 than negative passage embeddings. this paradigm does not provide explicit signals for understanding the relationships between questions, which causes the generated question embeddings to be insensitive to minimal discrepancies. as a result, the model generates overly similar embeddings for the meq and the original question, leading to incorrect passage retrieval for the meq. in fact, the overlap between the retrieved passages of the original question and those of its meq is as high as ∼70%, which reflects dpr's limited ability in distinguishing the questions. to overcome such limitations, it is necessary to complement dpr training with signals on inter-question relationships. besides building the mapping between questions and passages, dpr needs to know which questions are the same and which are different. in this pioneering study, we propose a simple and effective method based on a query-side contrastive loss to improve the performance of dpr on meqs. specifically, in order to learn inter-question relationships, dpr is trained to distinguish between paraphrase questions and semantically different questions. to achieve this, we obtain synthetic meqs for training questions from the machine-created qa corpus, paq @xcite , as augmented data. experiments demonstrate that learning the query-side contrastive loss on the augmented meqs improves the performance of dpr on contrast sets, without sacrificing its performance on standard open-domain questions in the nq test set. ## experiments in experiments, we compare our proposed training method against the original training setting of dpr. after training the models on the nq training set, we test them on the standard nq test set as well as two meq contrast sets that we collected in this work. ## conclusion in this study, we addressed the gap in research on contrast consistency in openqa by collecting meqs as challenging contrast sets to the popular nq benchmark. our findings reveal that dpr lacks contrast consistency on our contrast sets. to address this limitation, we introduced a query-side contrastive loss with the aid of data augmentation, which improved its ability to recognize inter-question relationships. overall, our findings and data can pave the way for further exploring the role of contrast consistency in developing robust and effective openqa systems.