Phrase Retrieval for Open Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning (2023)

Abstract: Open-Domain Conversational Question Answering (ODConvQA) aims at answering questions through a multi-turn conversation based on a retriever-reader pipeline, which retrieves passages and then predicts answers with them. However, such a pipeline approach not only makes the reader vulnerable to the errors propagated from the retriever, but also demands additional effort to develop both the retriever and the reader, which further makes it slower since they are not runnable in parallel. In this work, we propose a method to directly predict answers with a phrase retrieval scheme for a sequence of words, reducing the conventional two distinct subtasks into a single one. Also, for the first time, we study its capability for ODConvQA tasks. However, simply adopting it is largely problematic, due to the dependencies between previous and current turns in a conversation. To address this problem, we further introduce a novel contrastive learning strategy, making sure to reflect previous turns when retrieving the phrase for the current context, by maximizing representational similarities of consecutive turns in a conversation while minimizing irrelevant conversational contexts. We validate our model on two ODConvQA datasets, whose experimental results show that it substantially outperforms the relevant baselines with the retriever-reader. Code is available at: https://github.com/starsuzi/PRO-ConvQA .

## introduction conversational question answering (convqa) is the task of answering a sequence of questions that are posed during information-seeking conversations with users @xcite @xcite . this task has recently gained much attention since it is similar to how humans seek and follow the information that they want to find. to solve this problem, earlier convqa figure 1: (a) conventional retriever-reader pipeline approach, which first retrieves a relevant passage to a current conversational (i.e., conv.) context, and then predicts an answer based on the passage. (b) our direct phrase retrieval approach that predicts start and end tokens of the answer phrase based on their representational similarities to the current conv. context. to reflect the previous history when retrieving the phrase, we maximize representations of two consecutive conversations. work proposes to predict answers based on both the current question and the previous conversational histories, as well as the passage that is relevant to the ongoing conversation @xcite @xcite . however, this approach is highly suboptimal and might not be applicable to real-world scenarios, since it assumes that the gold passage, containing answers for the current question, is given to the convqa system; meanwhile, the gold passage is usually not available during the real conversation. to address this limitation, some recent work @xcite @xcite @xcite proposes to extend the existing convqa task to an opendomain question answering setting with an assumption that the conversation-related passages are not given in advance; therefore, it is additionally required to access and utilize the query-relevant passages in a large corpus, for example, wikipedia. under this open-domain setting, most existing open-domain convqa (odconvqa) work relies on the retriever-reader pipeline, where they first retrieve the passages, which are relevant to both the current question and conversational context, from a large corpus, and then predict answers based on information in the retrieved passages. this retrieverreader pipeline approach is illustrated in figure 1 . however, despite their huge successes, such a pipeline approach consisting of two sub-modules has a few major drawbacks. first, since the reader is decomposed from the retriever, it is difficult to train the retriever-reader pipeline in an end-to-end manner, which results in an additional effort to develop both the retriever and the reader independently. second, the error can be accumulated from the retriever to the reader, since the failure in finding the relevant passages for the current question negatively affects the reader in predicting answers, which is illustrated in figure 1 . third, while the latency is an important factor when conversing with humans in the real-world scenarios, the retrieverreader pipeline might be less efficient, since these two modules are not runnable in parallel. an alternative solution tackling the limitations above is to directly predict the phrase-level answers consisting of a set of words, which are predicted from a set of documents in a large corpus. while this approach appears challenging, recent work shows that it is indeed possible to directly retrieve phrases within a text corpus based on their representational similarities to the input question @xcite . however, its capability of retrieving phrases has been studied only with single-turn-based short questions, and their applications to odconvqa, additionally requiring contextualizing the multi-turn conversations as well as effectively representing the lengthy conversational histories, have not been explored. to this end, in this work, we first formulate the open-domain convqa task, previously done with the two-stage retriever-reader pipeline, as a direct phrase retrieval problem based on a single dense phrase retriever. however, in contrast to the single-turn open-domain question answering task that needs to understand only a single question, the target odconvqa is more challenging since it has to comprehensively incorporate both the current question and the previous conversational histories in multi-turns. for example, as shown in figure 1 , in order to answer the question, " what happened in 2003", the model has to fully understand that the conversational context is related to the song, not the movie. while some work @xcite @xcite proposes to feed an odconvqa model the entire context consisting of the current question together with the conversational histories as an input, this naïve approach might be insufficient to solve the conversational dependency issue, which may lead to suboptimal performances in a phrase retrieval scheme. in order to further address such a conversational dependency problem, we suggest to enforce the representation of the current conversational context to be similar to the representation of the previous context. then, since two consecutive turns in a conversation are dependently represented in a similar embedding space, phrases that are relevant to both the current and previous conversational contexts are more likely to be retrieved, for the current question. to realize this objective, we maximize the representational similarities between the current conversational context and its previous contexts, while minimizing the representations between the current and its irrelevant contexts within the same batch via the contrastive learning loss, which is jointly trained with the dense phrase retriever. this is illustrated in figure 1 , where we force the representation of the current conversational turn to be similar to its previous turn. we refer to our proposed method as phrase retrieval for open-domain conversational question answering (pro-convqa). we validate our proposed pro-convqa method on two standard odconvqa datasets, namely or-quac @xcite and @xcite , against diverse odconvqa baselines that rely on the retriever-reader pipeline. the experimental results show that our pro-convqa significantly outperforms relevant baselines. furthermore, a detailed analysis demonstrates the effectiveness of the proposed contrastive learning strategy and the efficiency of our phrase retrieval strategy. our contributions in this work are threefold: ## method in this section, we first define the conversational question answering (convqa) task, and its extension to the open-domain setting: open-domain convqa (odconvqa) in section 3.1. then, we introduce our dense phrase retrieval mechanism to effectively and efficiently solve the odconvqa task, compared to the conventional retriever-reader pipeline approach, in section 3.2. last, we explain our novel conversational dependency modeling strategy via contrastive learning, in section 3.3. ## experimental setups in this section, we explain datasets, metrics, models, and implementation details. ## results and discussion in this section, we show the overall results and provide detailed analyses. ## conclusion in this work, we pointed out the limitations of the retriever-reader pipeline approach to odconvqa, which is prone to error propagation from the retriever, unable to run both sub-modules in parallel, and demanding effort to manage these two submodules, due to its decomposed structure. to address such issues, we formulated the odconvqa task as a dense phrase retrieval problem, which makes it possible to directly retrieve the answer based on its representational similarity to the current conversational context. furthermore, to model the conversational dependency between the current and its previous turns, we force their representations to be similar with contrastive learning, which leads to retrieving more related phrases to the conversational history as well as the current question. we validated our proposed pro-convqa on od-convqa benchmark datasets, showing its efficacy in effectiveness and efficiency.