Learning to Retrieve Engaging Follow-Up Queries (2023)

Abstract: Open domain conversational agents can answer a broad range of targeted queries. However, the sequential nature of interaction with these systems makes knowledge exploration a lengthy task which burdens the user with asking a chain of well phrased questions. In this paper, we present a retrieval based system and associated dataset for predicting the next questions that the user might have. Such a system can proactively assist users in knowledge exploration leading to a more engaging dialog. The retrieval system is trained on a dataset called the Follow-up Query Bank (FQ-Bank). FQ-Bank contains ~14K multi-turn information-seeking conversations with a valid follow-up question and a set of invalid candidates. The invalid candidates are generated to simulate various syntactic and semantic confounders such as paraphrases, partial entity match, irrelevant entity, and ASR errors. We use confounder specific techniques to simulate these negative examples on the OR-QuAC dataset. Then, we train ranking models on FQ-Bank and present results comparing supervised and unsupervised approaches. The results suggest that we can retrieve the valid follow-ups by ranking them in higher positions compared to confounders, but further knowledge grounding can improve ranking performance.FQ-Bank is publicly available at https://github.com/amazon-science/fq-bank .

## introduction state of the art open domain conversational voice assistants can help users accomplish a wide range of tasks, including: factoid question answering, playing music, adding items to personal lists, controlling smart home appliances, and booking transportation. however, the linear nature of dialog with existing voice assistant technology makes it challenging for users to discover and fully utilize the * work done during internship. full range of these capabilities. in addition, successful utilization often requires exact formulation of the request, which further hinders the experience. one recent approach to addressing these issues in the voice assistant domain involves predicting relevant follow-up queries in order to assist the user with accomplishing their latent goals. 1 relevant follow-up queries (fqs) for typical voice assistant scenarios can range from specific command and control tasks such as "what is the temperature in new york" followed by "what is the chance of rain in new york?", to more open-ended knowledge exploration, e.g. "what is the capital of croatia" followed by "what is the population of croatia?". once a valid fq has been identified, the system can proactively recommend it to reduce user's cognitive load, e.g. "would you like to know what the population of croatia is?". this exchange is illustrated in figure 1 . the user can then be engaged in follow-up dialogue without the need to ask redundant questions. follow-up queries can be identified by retrieving and ranking candidates from a question repository. ## related works previous studies on proactivity in conversational ai mostly focuses on response generation. follow-up question identification and generation approaches have been explored from different perspectives. for example, @xcite explored the task of identifying if the latest user utterance is a follow-up of the previous questions or it has a different new context. this is helpful for understanding the question context properly and give the correct response. they also derived a new dataset called lif from the quac dataset @xcite , where each data point contains a conversation history, a new utterance, a passage used to answer the previous questions, one valid follow-up, and one or two invalid follow-ups. however, this dataset is focused on passage-based question answering, and the confounder typology does not address issues found in the search engine based fq retrieval scenario (e.g. paraphrases, irrelevant entity substitution, etc). other works have focused on generating followup queries for extracting information from users. for example, @xcite proposed a knowledge-driven system for generating follow-up queries, but it targeted the generation of follow-up survey questions to extract information from humans. @xcite explored systems for asking follow-up queries to interview candidates to extract more relevant information. our proposed method is focused not on information extraction from users, but rather providing highly relevant additional information to the user. ## follow-up query bank for our initial study on identifying fqs, we created a search index of information-seeking questions regarding public facts, spoken by users of a commercial voice assistant. we then queried the search engine with different types of information-seeking questions and analyzed the top negative (i.e., not a suitable follow-up) search results and categorized them into a typology of confounders. using this typology, we set out the task of simulating a similar scenario on a public conversation dataset and did not use any voice assistant data anymore. we selected or-quac as the seed dataset as it provides multi-turn information-seeking dialogs on a particular topic. in the rest of this section, we will provide a brief overview of the confounders and or-quac dataset, followed by an overview of the simulation methodology of the confounders using or-quac. ## learning to identify relevant follow-up task formulation: given a dialog @xmath0, a 1 ), ...., (q l-1 , a l-1 ), (q l , a l )} of l turns and a randomly organized set of n candidate follow-up queries @xmath1, y - 2 , ..., y - n-1 }, the task is to model p (i|x), @xmath2, such that argmax i p (i|x) = y + . here, q is a question, a is an answer, y + is a positive follow-up example, and y -is a set of negative examples. in order to develop a follow-up question candidate ranker, we experiment with different unsupervised and supervised approaches as described below. ## conclusions in this paper, we sought to address the problem of identifying valid and engaging follow-up queries for a user interacting with a conversational assistant. we experimented with a retrieval and ranking based framework to achieve this using a search engine and a database of past queries. in doing so, we identified a typology of confounders returned by the search. in order to train a ranking model to identify valid follow-up queries, we synthetically generated confounders based on a publicly available conversation dataset. we showed that our approach of ranking retrieved candidates based on their validity as follow-up queries achieved reasonable performance, but also that integrating external knowledge on entities or topics could improve follow-up selection. we have made the dataset publicly available to enable further research in this direction.