Integrating Question Rewrites in Conversational Question Answering: A Reinforcement Learning Approach (2022)

Abstract: Resolving dependencies among dialogue history is one of the main obstacles in the research on conversational question answering (QA). The conversational question rewrites (QR) task has been shown to be effective to solve this problem by reformulating questions in a self-contained form. However, QR datasets are limited and existing methods tend to depend on the assumption of the existence of corresponding QR datasets for every CQA dataset. This paper proposes a reinforcement learning approach that integrates QR and CQA tasks without corresponding labeled QR datasets. We train a QR model based on the reward signal obtained from the CQA, and the experimental results show that our approach can bring improvement over the pipeline approaches.

## introduction conversational question rewrites (qr) systems paraphrase a question into a self-contained format using its dialogue history so as to make it easier to understand by the conversational question answering (cqa) system. prior works @xcite @xcite have shown that explicit guidance of qr benefits the performance of the cqa models in multiple questions answering datasets. however, the existing works on qr in the context of cqa are often ignorant of two critical issues. firstly, they are dependent on the assumption that qr datasets exist on target cqa datasets, although existing qr datasets only cover a small amount of cqa. it is also notable that building a novel qr dataset is expensive. current works mainly focus on quac @xcite datasets thanks to qr datasets constructed from it @xcite , however, the other popular ⇤ equal contribution cqa datasets such as coqa @xcite remain less explored. secondly, although qr task evaluation is mainly done by automatic metrics that compute n-gram overlaps with bleu @xcite or rouge @xcite , and by human evaluation, there is no correlation guaranteed between those metrics and the performance in cqa. in fact, petrén bach hansen and søgaard (2020) and @xcite suggest "better" rewrites in the human eye are not necessarily better for machines. to this end, we propose to alleviate the limitation of the qr system by introducing a reinforcement learning framework that utilizes qr to overcome the aforementioned two obstacles. in this framework, a qr model plays the role of "the agent" which receives rewards from a cqa model which acts as "the environment." during training, a qr model aims to maximize the performance on the cqa task by generating better rewrites of the questions. exploiting the reinforcement learning nature, we can benefit cqa regardless of the existence of qr annotation, and we can ensure that qr contributes to the final objective of improving cqa. experimental results show that our framework successfully improves the cqa performance by 4.1 to 8.6 f1 score on coqa and 4.7 to 9.2 f1 on quac compared to the pipeline baselines that combine a qr model and a qa model. our contributions in this paper can be summarized three-fold as follow: figure 1 : overview of our reinforcement learning approach for cqa task that involves a qr model and a qa model. the current question q t and its dialogue history are reformulated into a self-contained question qt by the qr model. then, qt and optionally, its dialogue history is passed to the qa model to extract an answer span ãt from the provided evidence document. we train the qr model by maximizing the reward signal (f1 score) obtained by the comparison between the predicted answer span ãt and the gold span a t . ## methodology in this section, we present our reinforcement learning framework and the training algorithm. firstly, we offer several preliminary definitions used throughout the paper, and secondly, we describe the strategy to train the whole framework. ## discussion in this section, we provide our findings regarding the most suitable settings of our approach, including the comparison of the qr datasets for initialization, the qr model architectures, the training algorithms, and the effect of the decoding strategy. for automatic evaluation, we report exact match (em) in addition to the unigram f1 score. em in-models # params evaluation f1 em gpt-2 243m 84.7 76.6 bart-base 210m 83.6 75.7 ## conclusion and future work in this paper, we propose a reinforcement learning framework for cqa that a qr model that acts as an agent and a qa model as an environment. our experiments show that the qr model learned to paraphrase questions into a more suitable format for the qa model by reward signal obtained from the cqa performance. since our exploration is conducted with limited combinations of qa/qr model structures and datasets, we plan to explore the other combinations to justify our approach. moreover, it would be beneficial to train the qr model without the dialogue history to enforce the qr model and make the question more self-contained. if we can minimize the contribution of the dialogue history in cqa, we can treat the cqa task as a singleturn qa task, and it enormously expands possible solutions for the cqa.