Strong and Efficient Baselines for Open Domain Conversational Question Answering (2023)

Abstract: Unlike the Open Domain Question Answering (ODQA) setting, the conversational (ODConvQA) domain has received limited attention when it comes to reevaluating baselines for both efficiency and effectiveness. In this paper, we study the State-of-the-Art (SotA) Dense Passage Retrieval (DPR) retriever and Fusion-in-Decoder (FiD) reader pipeline, and show that it significantly underperforms when applied to ODConvQA tasks due to various limitations. We then propose and evaluate strong yet simple and efficient baselines, by introducing a fast reranking component between the retriever and the reader, and by performing targeted finetuning steps. Experiments on two ODConvQA tasks, namely TopiOCQA and OR-QuAC, show that our method improves the SotA results, while reducing reader’s latency by 60%. Finally, we provide new and valuable insights into the development of challenging baselines that serve as a reference for future, more intricate approaches, including those that leverage Large Language Models (LLMs).

## introduction in an automated information-seeking conversation scenario between two parties, the human questioner asks a series of questions and expects to receive a relevant response from the answering system @xcite . current state-of-the-art (sota) shapes the answerer via two neural models, the dense passage retrieval (dpr) @xcite and the fusion-in-decoder (fid) @xcite , which act as retriever and reader, respectively. their success stems from the ability to overcome certain limitations of their sparse and extractive counterparts, such as not relying on lexical retrieval heuristics or extracting spans as a response @xcite @xcite @xcite @xcite . among the most promising approaches are those concerning the improvement of training strategies @xcite @xcite , use of rerankers @xcite @xcite @xcite , question rewriting @xcite , reader to retriever knowledge distillation @xcite , memory-efficient pipeline @xcite , and leveraging structured information @xcite . unlike the open domain question answering (odqa) setting, a reassessment of the baselines in terms of both efficiency and effectiveness appears to be under-explored in the conversational (od-convqa) domain. in this paper, we focus on the typical dpr retriever and fid reader (dpr+fid) pipeline, and show its limitations when applied to the odconvqa setting. despite its popularity, we find that this baseline significantly underperforms when finetuned on downstream tasks. we show that simple improvements in the training, architecture, and inference setups of the dpr+fid pipeline, provide a strong and efficient baseline that exceeds the performance of sota models on two common odconvqa datasets: topiocqa @xcite and orconvqa (or-quac) @xcite . we point out several limitations of the pipeline, such as: 1) reader's susceptibility to noisy input, 2) retriever's reduced coverage, 3) retriever's lack of cross semantic encoding between the conversation and the retrieved passages, and 4) reader's latency is heavily impacted by the number of input passages. to mitigate these, we propose and evaluate a simple and effective approach by including a fast reranking component between the retriever and the reader, and by performing targeted finetuning steps. the proposed retriever-reranker-reader finetuning (r3fine) strategy leads to baseline models with a better latency/performance trade-off. these baselines, which are simple and easy to replicate, serve as a reference point for comparing new and more complex models, and determining their effectiveness. our contributions are the following: this section provides a brief introduction to the pipeline on which this work focuses. figure 1 shows the typical pipeline used within the od-convqa setting, featuring an additional reranker component. a conversation history is input to the dpr retriever. this module exploits a dualencoder based on the bert @xcite model. first, it encodes the conversation history via the conversationencoder component, which takes as input the text of the conversation history c 1 , c 2 , . . . , c i , and then it outputs a dense representation h c . next, this representation is used to perform a dense search to retrieve the most relevant passages, i.e., text blocks that serve as basic retrieval units, from an external knowledge source (e.g., wikipedia). the latter contains dense representations of the passages that have been encoded via the p assageencoder component, which takes as input a j-th passage with a given text length n , i.e., p j 1 , p j 2 , . . . , p j n , and outputs a dense representation h p j . the dense search is performed via the maximum inner-product search (mips) function which outputs the value corresponding to h ⊺ c • h p j . once the top-k relevant passages have been retrieved, their text is appended to the conversa-tion history and subsequently passed to the fid reader, which is based on the t5 @xcite model. the newly created textual sequences of length s are then encoded in parallel via the encoder component that outputs a dense representation @xmath0, . . . , h s }. as a final step, the dense representations of the entire list of k input passages are concatenated to form a single h1 ⊕ h2 , . . . , ⊕ hk sequence that forms the input to the decoder component responsible for generating the answer a. ## strong baseline models this work focuses on two main datasets. topiocqa @xcite is a large-scale open-domain information-seeking conversational dataset that contains a challenging phenomenon in the form of topic switching. or-quac @xcite ) leverages canard's @xcite context-independent question rewrites of the quac @xcite dataset, and adapts it to the open-domain setting. further details regarding the datasets are provided in appendix a. we outline a number of limitations of the typical dpr+fid pipeline, along with suggestions on how to mitigate them. while some of those interconnect at different levels the various efforts made in the odqa domain @xcite , our goal is to offer a perspective on the odconvqa setting. ## experiments and results this section shows the impact that the introduction of the semanticreranker module has on the pipeline, as well as the finetuning steps we followed to make the pipeline more efficient without compromising its performance. ## conclusions in this paper, we identified several limitations of the typical depnse passage retrieval (dpr) retriever and fusion-in-decoder (fid) reader pipeline when applied in an odconvqa setting. we proposed and evaluated an improved approach by including a fast reranking component between these two modules and by performing targeted finetuning steps. the proposed r3fine strategy lead to a better latency/performance trade-off. the new baseline has proven to be both strong and efficient when compared to previous baselines, thus making it suitable for future comparisons of new approaches.