Reinforced Question Rewriting for Conversational Question Answering (2022)

Abstract: Conversational Question Answering (CQA) aims to answer questions contained within dialogues, which are not easily interpretable without context. Developing a model to rewrite conversational questions into self-contained ones is an emerging solution in industry settings as it allows using existing single-turn QA systems to avoid training a CQA model from scratch. Previous work trains rewriting models using human rewrites as supervision. However, such objectives are disconnected with QA models and therefore more human-like rewrites do not guarantee better QA performance. In this paper we propose using QA feedback to supervise the rewriting model with reinforcement learning. Experiments show that our approach can effectively improve QA performance over baselines for both extractive and retrieval QA. Furthermore, human evaluation shows that our method can generate more accurate and detailed rewrites when compared to human annotations.

## introduction interacting through conversations is a natural information-seeking procedure for humans, therefore it is important for ai assistants like apple siri and amazon alexa to enable and improve such experiences. in recent years conversational question answering (cqa) has gained more attention, where a user can ask a series of related questions and ideally obtain answers that leverage the conversational context. different from widelystudied question answering (qa) tasks that happen in single-turn @xcite @xcite , the interpretation of conversational questions in cqa depends on questions and answers from previous turns. previous approaches to cqa usually train new models from scratch, which can achieve promising results but also are expensive in terms of obtaining domain-specific training data. in industry settings, there are many single-turn qa models deployed. training new cqa models with additional annotations to replace each existing single-turn qa model is expensive, and generally not feasible. moreover, discarding existing single-turn models and datasets is impractical, and studying how to reuse these existing resources to tackle cqa merits attention. existing approaches to this task, called conversational question rewriting (cqr), often train sequence-to-sequence models supervised by human rewrites to generate self-contained questions @xcite . such methods have several limitations. first, the cqr training objective is disconnected from cqa performance. the annotation process of existing rewriting datasets has no knowledge of the qa systems, and more human-like rewrites do not guarantee better cqa performance. second, the rewriting model does not take into account the feedback from downstream qa systems. in industry settings, multiple single-turn qa systems trained with different datasets serve in the backend. it is impractical to replace them with new cqa models, and we argue that their output can still be used as signals to help train rewriting models. to overcome these limitations, we propose an effective cqr approach upon the recent success of reinforcement learning (rl) techniques for text generation @xcite . rl enables flexible ways to incorporate training objectives in the form of reward functions. we systematically analyze different rewards and their effectiveness in terms of final qa performance, as well as the quality of the question rewrites (i.e. the question still has to be understandable and interpretable by humans). to optimize qa performance, we propose various qa rewards to measure the likelihood of a question yielding a better answer. in comparison with the qa rewards, we also propose to use the same rl approach with question rewriting (qr) rewards reflecting the similarity between a modelgenerated question and the human's ground-truth. we summarize our contributions as follows: conversational question answering. recently, conversational qa has been studied which presents new challenges for qa models such as being able to resolve conversational dependencies so that a conversational question can be interpreted by qa models with conversational context. quac (elgohary et al., 2019) and coqa (reddy et al., 2019) are two datasets for extractive cqa where answers are extracted from passages. cast-19 (dalton et al., 2020) is a benchmark for retrieval cqa and the target is to return relevant passages given a question. qrecc (anantha et al., 2021) combines retrieval and extractive cqa where the answers are extracted from passages returned by a retrieval component. kim et al. (2021) propose to train the cqa model and rewriter simultaneously, which is impractical for industry setting. a directly related work to ours is vakulenko et al. (2021) which proposes to rewrite questions for cqa. however, they do not consider taking the qa feedback into the cqr training which is studied in our work. rl for nature language generation. reinforcement learning methods have been widely applied for various language generation tasks. li et al. (2016) propose to apply deep reinforcement learning in dialogue generation to model future rewards related to conversational properties, such as informativeness, coherence and ease of answering. ranzato et al. (2016) propose mixed incremental cross-entropy reinforce (mixer) for sequence prediction to directly optimize the metrics used at test time, such as bleu or rouge. they show mixer outperforms several strong baselines for greedy generation on text summarization, image caption and machine translation. nogueira and cho (2017) train a query rewriter based on the rewards relying on the ground-truth ranking list for information retrieval. buck et al. (2018) use rl for single-turn question rewriting by maximizing the answers' quality which requires ground-truth. similar to our f1 reward, wu et al. (2021) design rewards from ground-truth answers to train a conversational query rewriter. instead, we propose alternative rewards indicating the confidence of answers from a model itself which do not require human annotations. ## problem definition in cqa, each conversation contains a sequence of (question, answer) pairs @xmath0, a 1 , ..., q n , a n }, where a i is the answer for question q i . a conversational question q i can be ambiguous and its interpretation depends on the conversational context c @xmath1, a 1 , ..., q i-1 , a i-1 }. the goal of cqr for qa is to learn a model r θ , parameterized by θ, that can translate q i associated with c i into q ′ i , so that the semantic meaning of q ′ i is equivalent to q i . a pretrained single-turn qa model is expected to answer q ′ i better than q i . note that the qa model can be trained from a single-turn dataset different from d and is fixed when training the rewriter. the motivation is to explore whether the already deployed single-turn qa models can be exploited to train a rewriter and reused without further training by accepting the rewritten questions. ## results here, we study the following research questions: rq1: can our proposed qr and qa rewards improve the overall cqa performance? in particular, how effective are unsupervised rewards? rq2: does achieving the best qr target mean achieving the best qa target? rq3: what is the quality, as judged by humans, of the reward-guided question rewrites? ## conclusion we proposed a conversational question rewriting (cqr) approach using reinforcement learning. such rewriting approaches are an emerging solution in real-world settings where qa systems with many existing answering backends trained on standalone questions must be adapted to work in conversational settings. after assessing various qa and qr rewards, we showed that optimizing qr rewards is limited in improving cqa performance. in contrast, qa rewards that do not require ground-truth annotations consistently achieve the best cqa performance over baselines. for extractive cqa, using confidence rewards improved f1 by 2% over bart-based baseline on canard; and for retrieval cqa, using bm25 rewards improved the ndcg@3 of the baseline by 3.4% on cast-19. a human evaluation also demonstrated that our approach can generate higher-quality rewrites with more accurate and detailed context information.