Learning to Identify Follow-Up Questions in Conversational Question Answering (2020)

Abstract: Despite recent progress in conversational question answering, most prior work does not focus on follow-up questions. Practical conversational question answering systems often receive follow-up questions in an ongoing conversation, and it is crucial for a system to be able to determine whether a question is a follow-up question of the current conversation, for more effective answer finding subsequently. In this paper, we introduce a new follow-up question identification task. We propose a three-way attentive pooling network that determines the suitability of a follow-up question by capturing pair-wise interactions between the associated passage, the conversation history, and a candidate follow-up question. It enables the model to capture topic continuity and topic shift while scoring a particular candidate follow-up question. Experiments show that our proposed three-way attentive pooling network outperforms all baseline systems by significant margins.

## introduction conversational question answering (qa) mimics the process of natural human-to-human conversation. recently, conversational qa has gained much attention, where a system needs to answer a series of interrelated questions from an associated text passage or a structured knowledge graph @xcite @xcite . however, most conversational qa tasks do not explicitly focus on requiring a model to identify the follow-up questions. a practical conversational qa system must possess the ability to understand the conversation history well, and to identify whether the current question is a follow-up of that particular conversation. consider a user who is trying to have a conversation with a machine (e.g., siri, google home, alexa, cortana, etc). first, the user asks a question and the machine answers it. when passage: . . . script for verhoeven's first american film, @xcite , which starred rutger hauer and jennifer jason leigh. verhoeven moved to hollywood for a wider range of opportunities in filmmaking. working in the u.s. he made a serious change in style, directing big-budget, very violent, special-effectsheavy smashes robocop and total recall. robocop, for . . . verhoeven followed those successes with the equally intense and provocative basic @xcite . . . received two academy awards nominations, for film editing and for original music . . . conversation history: q: what was the first film verhoeven did in the us? a: flesh and blood q: what genre of films did he make? a: big-budget, very violent, special-effects-heavy smashes candidate follow-up question examples: what year did his first film debut? -valid did he make any films during his final years? -invalid what did she do after her debut film? -invalid the user asks the second question, it is very important for the machine to understand whether it is a follow-up of the first question and its answer. further, this needs to be determined for every question posed by the user in that ongoing conversation. by identifying whether the question is a follow-up question, a machine determines whether the conversation history is relevant to the question. based on this decision, it is expected to use a suitable answer finding strategy for answering the question. additionally, a qa system first retrieves some relevant documents using an information retrieval (ir) engine to answer a question. if a follow-up question identifier predicts the question as an invalid followup question given the retrieved documents, it can communicate to the ir engine to retrieve additional supporting documents. a few example instances are given in figure 1 to illustrate the follow-up question identification task in a conversational reading comprehension setting. we present a new dataset for learning to identify follow-up questions, namely lif. given a text passage as knowledge and a series of question-answer pairs as conversation history, it requires a model to identify whether a candidate follow-up question is valid or invalid. the proposed dataset requires a model to understand both topic continuity and topic shift to correctly identify a follow-up question. for instance, in the first example given in figure 1 , a model needs to capture the topic continuity from the first question-answer pair (i.e., first film is flesh and blood) and the topic shift from the second question-answer pair (i.e., genre of films) of the conversation history. the candidate followup question in the second example is invalid since the associated passage does not provide any information about his final years. the last follow-up question example is invalid since verhoeven is a he, not she. there has been some research in the past which focuses on identifying what part of the conversation history is important for processing follow-up questions @xcite . however, the recently proposed neural network-based models for conversational qa have not explicitly focused on follow-up questions. in this paper, we propose a three-way attentive pooling network for follow-up question identification in a conversational reading comprehension setting. it evaluates each candidate follow-up question based on two perspectives -topic shift and topic continuity. the proposed model makes use of two attention matrices, which are conditioned over the associated passage, to capture topic shift in a follow-up question. it also relies on another attention matrix to capture topic continuity, directly from the previous question-answer pairs in the conversation history. for comparison, we have developed several strong baseline systems for follow-up question identification. the contributions of this paper are as follows: given a passage, a sequence of question-answer pairs in a conversation history, and a candidate follow-up question, the task is to identify whether or not the candidate follow-up question is a valid follow-up question. we denote the passage as p which consists of t tokens. let the sequence of previous questions and their corresponding answers be denoted as {q 1 , q 2 , . . . , q m } and {a 1 , a 2 , . . . , a m }, where m is the number of previous question-answer pairs in the conversation history. the candidate follow-up question is denoted as c. we formulate this task as a binary classification task, which is to classify c as valid or invalid. in the remainder of this paper, we denote the length of the candidate follow-up question as v . in our model, we concatenate all previous questions and their answers with special separator tokens as follows: the combined length of the previous questionanswer pairs in the conversation history is denoted as u . ## lif dataset in this section, we describe how we prepared the lif dataset, followed by an analysis of the dataset. ## three-way attentive pooling network in this section, we describe our proposed three-way attentive pooling network foot_0 . first, we apply an embedding layer to the associated passage, the conversation history, and the candidate follow-up question. further, they are encoded to derive sequence-level encoding vectors. then the proposed three-way attentive pooling network is applied to score each candidate follow-up question. ## baseline models we develop several rule-based, statistical machine learning, and neural baseline models. for all the models, a threshold is determined based on the best performance on the development set. ## experiments in this section, we present the experimental settings, results, and performance analysis. ## related work many data-driven machine learning methods have been shown to be effective for tasks relevant for dialog such as dialog policy learning @xcite , dialog state tracking @xcite @xcite , and natural language generation @xcite @xcite . most of the recent dialog systems are either not goal oriented (e.g., simple chit-chat bots), or domain-specific if they are goal oriented (e.g., it help desk). in the last few years, there has been a surge of interest in conversational question answering. @xcite released a complex sequential question answering (csqa) dataset for learning conversations through a series of interrelated qa pairs by inferencing over a knowledge graph. @xcite released a large-scale conversational qa dataset, namely question answering in context (quac), which mimics a student-teacher interactive scenario. @xcite released the coqa dataset and many systems were evaluated on it. @xcite proposed sdnet to fuse context into traditional reading comprehension models. huang et al. (2019) proposed a "flow" mechanism that can incorporate intermediate representations generated during the process of answering previous questions, through an alternating parallel processing structure. in a conversation setting, given the previous qa pairs as conversation history, while these models focus on answering the next question, our work is focused on identifying follow-up questions. recently, @xcite proposed a dataset for regulatory texts that requires a model to ask follow-up clarification questions. however, the answers are limited to yes or no, which makes the task rather passage: on september 9, 1971, prisoners at the state penitentiary at attica, ny, took control of a cell block and seized thirty-nine correctional officers as hostages. after four days of negotiations, department of correctional services commissioner russell oswald agreed to most of the inmates' demands for various reforms but refused to grant complete amnesty to the rioters, with passage out of the country and removal of the prison's superintendent. when negotiations stalled and the hostages appeared to be in imminent danger, rockefeller ordered new york state police and national guard troops to restore order and take back the prison on september 13. . . . history: q: where was the attica prison? a: on september 9, 1971, prisoners at the state penitentiary at attica, ny, took . . . q: why did they riot? a: the inmates' demands for various reforms but refused to grant complete amnesty to the rioters, with passage out of the country and removal of the prison's superintendent. candidate follow-up question: how long did the riot last? -valid passage: in 1975, at age 22, schuur auditioned for drummer/bandleader ed shaughnessy. escorted by her twin brother, she went backstage to . . . singing the blues. . . . he hired her to be the vocalist in his orchestra, "energy force". jazz ## conclusion in this paper, we present a new follow-up question identification task in a conversational setting. we developed a dataset, namely lif, which is derived from the previously released quac dataset. notably, the proposed dataset supports automatic evaluation. we proposed a novel three-way attentive pooling network which identifies whether a follow-up question is valid or invalid by considering the associated knowledge in a passage and the conversation history. additionally, we developed several strong baseline systems, and showed that our proposed three-way attentive pooling network outperforms all the baseline systems. incorporating our three-way attentive pooling network into open domain conversational qa systems will be interesting future work.