Making Information Seeking Easier: An Improved Pipeline for Conversational Search (2020)

Abstract: This paper presents a highly effective pipeline for passage retrieval in a conversational search setting. The pipeline comprises of two components: Conversational Term Selection (CTS) and Multi-View Reranking (MVR). CTS is responsible for performing the first-stage of passage retrieval. Given an input question, it uses a BERT-based classifier (trained with weak supervision) to de-contextualize the input by selecting relevant terms from the dialog history. Using the question and the selected terms, it issues a query to a search engine to perform the first-stage of passage retrieval. On the other hand, MVR is responsible for contextualized passage reranking. It first constructs multiple views of the information need embedded within an input question. The views are based on the dialog history and the top documents obtained in the first-stage of retrieval. It then uses each view to rerank passages using BERT (fine-tuned for passage ranking). Finally, MVR performs a fusion over the rankings produced by the individual views. Experiments show that the above combination improves first-state retrieval as well as the overall accuracy in a reranking pipeline. On the key metric of NDCG@3, the proposed combination achieves a relative performance improvement of 14.8% over the state-of-the-art baseline and is also able to surpass the Oracle.

## introduction the abilities of current conversational assistants @xcite to perform open-domain conversational information seeking (cis) functions are limited @xcite . thus, to encourage and support research on conversational information seeking, the trec conversational assistance track (cast) @xcite defined a model title: goat breeds description: interested in buying goats that implies interest in different breeds of goats and their use (milk, meat, and fur). ## related work previous research provides guidance about the requirements of conversational search systems. for example, @xcite described desirable key features for conversational information retrieval systems. trippas et al. (2018) identified commonly-used interactions and informed conversational search system design by studying the conversations of real users. @xcite released the microsoft information-seeking conversation (misc) dataset, which mimics conversational assistants such as cortana. prior to the cast dataset, researchers often utilised dialog response reranking tasks @xcite , conversational questionanswering @xcite and voice based recommendation @xcite as a 'proxy' for a conversational search setting. for example, @xcite presented an end-to-end trainable attentive memory network for reading comprehension. @xcite proposed a method for dialog response ranking that incorporates external knowledge into deep neural models with pseudo-relevance feedback. @xcite formulated the task of asking clarifying questions in open-domain information-seeking conversational systems. the introduction of the cast dataset @xcite has brought in a new range of systems which focus on conversational information seeking. the ateam's run @xcite of trec cast 2019 utilises gpt-2 @xcite to translate questions augmented with previous turns of the conversation into stand-alone questions that are afterwards used to retrieve relevant passages. their question rewriting approach is based on a transfer learning paradigm. on the other hand, to overcome the problem of limited data, @xcite propose two ad-hoc approaches based on historical question expansion and historical answer expansion in combination with bert @xcite for ranking passages. more recently, yu et al. (2020) utilise rule-based and self-supervised methods to generate weak supervision data using large amounts of ad hoc search sessions which in previous questions (conversation history) current question term selection classifier search engine passages multi-view queries reranked passages conversation conversational term selection multi-view reranking bert reranker fusion turn are used to fine-tune gpt-2 in order to rewrite conversational queries. the rewritten queries are then used for ranking passages. ## proposed approach figure 1 presents an overview of the approach. the pipeline consists of i) conversational term selection (cts), and ii) multi-view reranking. the cts component uses a classifier to select relevant contextual terms from dialog history. the classifier's predictions are then used to convert the given question into a query before submitting it to the search engine. the top r passages obtained with this method are passed to the multi-view reranking component which begins by projecting the input question into multi-view queries. the first view is based on question reformulation, the second view utilizes the cts predictions, whereas the third view uses the top retrieved passages obtained using cts. each of these views are then individually used for reranking. the process of reranking is performed using a bert-based reranker. finally, the rerankings produced using the individual views are combined using fusion. cts and mvr are described in details below. ## experimental methodology dataset: the cast dataset @xcite consists of 30 training topics (9 questions per topic, 269 in total), and 50 test topics (9.6 questions per topic, 478 in total). however, relevance judgments are available only for 20 test topics (173 questions). therefore, evaluation is performed only over the 20 judged topics. the passages in cast dataset are borrowed from msmarco and trec complex answer retrieval track. on the other hand, the annotated camrest676 dataset, which is used for weak supervision, consists of 676 dialogs with coreferences and ellipsis annotations @xcite . parameter settings: the cts classifier uses bert-base-uncased model and is fine-tuned for 5 epochs. it uses adam (kingma and ba, 2014) as the optimiser with a learning rate of 5 × 10 -5 . while training, the maximum length of the context is clipped to 100, and the length of the input question is clipped to 30. on the other hand, mvr uses a bert-base-uncased model fine-tuned on 2% of the ms-marco passage ranking dataset. during training, the maximum length of the query is clipped to 64, whereas that of the passage is clipped to 256. the first-round of retrieval by cts leads to a total of 1000 passages per input question. during the reranking phase, only top r of the initial passages are reranked by mvr. evaluation metrics: the performance of the cts classifier is measured using precision (prec), recall and f1. the passage retrieval performance is measured using normalized discounted cumulative gain at a ranking depth of 3 (ndcg@3) which is the main metric prescribed by trec cast. the results are also evaluated using mean reciprocal rank (mrr). ## experiments and results this section is divided into two halves. the first half evaluates the performance of cts. the second half evaluates the performance of mvr i.e the result of using the entire pipeline. ## conclusion this paper presents a simple yet highly effective pipeline for conversational search. the pipeline consists of two components: cts and mvr. cts aids in first-round of passage retrieval by selecting important contextual terms from the dialog history. mvr reranks the passages obtained by cts by expressing the information need embedded within a question in multiple forms. the combination is able to surpass the state-of-the-art and at the same time perform slightly better than the oracle. to the best of our knowledge, no automatic system has been able to do so.