Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning (2020)

Abstract: The interaction of conversational systems with users poses an exciting opportunity for improving them after deployment, but little evidence has been provided of its feasibility. In most applications, users are not able to provide the correct answer to the system, but they are able to provide binary (correct, incorrect) feedback. In this paper we propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. We perform simulated experiments on document classification (for development) and Conversational Question Answering datasets like QuAC and DoQA, where binary user feedback is derived from gold annotations. The results show that our method is able to improve over the initial supervised system, getting close to a fully-supervised system that has access to the same labeled examples in in-domain experiments (QuAC), and even matching in out-of-domain experiments (DoQA). Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.

## introduction in conversational question answering (cqa) systems, the user makes a set of interrelated questions to the system, which extracts the answers from reference text @xcite . these systems are trained on datasets of human-human dialogues collected using wizard-of-oz techniques, where two crowdsourcers are paired at random to emulate the questioner and the answerer. several projects have shown that it is possible to train effective systems using such datasets. for instance, quac includes question and answers about popular people in wikipedia @xcite , and doqa includes question-answer conversations on cooking, movies and travel faqs @xcite . building such datasets comes at a cost, which limits the widespread use of conversational systems built using supervised learning. the fact that conversational systems interact naturally with users poses an exciting opportunity to improve them after deployment. given enough training data, a company can deploy a basic conversational system, enough to be accepted and used by users. once the system is deployed, the interaction with users and their feedback can be used to improve the system. in this work we focus on the case where a cqa system trained off-line is deployed and receives explicit binary (correct, incorrect) feedback from users. an example of this task can be seen in figure 1 where at a point in the conversation two different users give binary feedback to the system according to the correctness of the received answer. assuming a large number of interactions, we can safely ignore examples for which no feedback is received. we propose feedback-weighted learning (fwl) based on importance sampling as the technique to improve the initial supervised system using only binary feedback from users. in our experiments user feedback is simulated, and the correct/incorrect feedback is extracted from the gold standard. that is, if the system output matches the gold standard output then it is deemed correct, otherwise it is taken to be incorrect. in order to develop and test feedback-weighted learning we perform initial experiments on document classification. the results show that the model improved by the user 2 figure 1 : example of the cqa task where at a point in the conversation the user 1 gives positive feedback to the system and user 2 gives a negative one due to the received incorrect answer. regarding cqa, we use the best hyperparameters from the earlier experiment on document classification, and conduct experiments using several domains in cqa including datasets like quac and doqa. our method always improves over the initial supervised system. in the in-domain experiments (quac) our method is close to the fully supervised model which is fine-tuned with true labels rather than binary feedback, and in the out-of-domain experiments (doqa) our method matches it. the out-of-domain results are particularly exciting, as they are related to the case where a cqa system trained off-line in one domain could be deployed in another domain, letting the users improve it via their partial feedback by interacting with the system. our experiments reveal that the proposed approach is robust to the choice of the system architecture, as we experimented with both multi-layer perceptron and pre-trained transformer. the main contribution of our work is a novel method based on importance sampling, feedbackweighted learning, which improves the results of two widely used deep learning architectures using partial feedback only. experimental results from document classification show that feedback-weighted learning improves over the initial supervised system, matching the performance of a fully supervised system which uses true labels. in-domain and out-of-domain cqa experiments show that the proposed method improves over the initial supervised system in all cases, matching a fully supervised system in out-of-domain experiments. this work opens the prospect to exploit interactions with real users and improve conversational systems after deployment. all the code and dataset splits are made publicly available foot_0 . ## related work on conversational question answering cqa research builds on reading comprehension. in reading comprehension the system has to answer questions about a certain passage of text in order to show that it understands the passage. there are two main methods: the extractive method, in which the answer is selected as a contiguous span in the reference passage, and the abstractive method, in which the answer text is generated. many datasets @xcite @xcite @xcite and systems have been proposed to address this task, where the extractive scenario has drawn special attention @xcite . lately, with the incursion of large pre-trained language models as bert @xcite , xlnet @xcite and their relatives, the state of the art has been dominated by systems that use the representations obtained with these pre-trained language models. the systems learn answer pointer networks that consist of two classifiers, one for spotting the start token of the answer span and another for spotting the end token of the answer span. in reading comprehension, the questions are individual and isolated, that is, they do not have any dialogue structure. due to the increasing interest on modelling the conversational structure behind user questions, several cqa datasets where questions and answers are interrelated have been created following the wizard-of-oz technique. among all the datasets we can highlight quac @xcite , coqa @xcite and @xcite . while the first two datasets cover more formal domains as wikipedia articles and literature, the latter covers different domains extracted from online forums as stackexchange. contextual versions of the previously mentioned reading comprehension models have successfully modelled the conversational structure in those datasets @xcite @xcite . ## importance sampling for learning after deployment in our learning after deployment scenario we start by training an initial s 0 system in an off-line and supervised way. this first system follows the traditional workflow where we have access to limited supervised training and development data. then, we take the best performing system on the development data and deploy it to serve user queries. in this deployment phase, every time a user makes a query x, the system generates an answer y and the user gives binary feedback to it. over time, the system generates different answers y i1 , y i2 , ..., y in and receives feedback for each item x i . we assume a sufficient amount of user interactions, and as such we ignore any query-answer pair for which the user did not provide feedback. after the system has been deployed for a while, we collect for each question the answers provided by the system, and the respective user feedback. we consider a cqa system implemented using two classifiers predicting the start and end tokens respectively. this allows us to consider each classifier independently and describe the process of learning after deployment for a single classifier. we propose to use feedback-weighted learning, which is based on self-normalized importance sampling, in order to generate the system answers. ## experiments in this section we present the experiments with feedback-weighted learning (fwl). in the experiments we first build a supervised system (s 0 ), and then we simulate a deployment phase by letting s 0 answer user queries and receiving their feedback. user feedback is derived from a manually annotated deployment set, which is obtained by splitting the training set. we refer to the set used for training s 0 as a training set and the other partition of the original training set as a deployment set in the rest of the paper. we consider the following systems and baselines: although our main objective is to develop a lifelong learning system for cqa, we also perform experiments on document classification, as a way to assess the robustness of the proposed method when applied to different neural architectures and tasks. moreover, these experiments are used to develop the system and check the impact of hyperparameters, so that the best hyperparameters from document classification are used in the cqa experiments. ## discussion as shown by the experiments in document classification and cqa we are able to improve an initial supervised s 0 system just by using binary feedback obtained by simulating the users. in this section we perform a further analysis on several aspacts of the method. hyperparameters. in order to show the robustness of fwl we perform several experiments in the document classification task with different values for the main hyperparameters of the method, λ and β (cf. section 3.1). the analysis shows that when using values larger than 1 for β, fwl performs similarly well for all lambdas greater than 0.8 (see figures 2a and 2b ). the behavior of λ in the same figures 2a and 2b reveals that large values of λ yields best results for all beta values. in any case, the similar performance obtained with different hyperparameter combinations shows that our method is robust and not specially sensitive to small variations in the hyperparameters. ## conclusion and future work in this work we propose feedback-weighted learning that allows a supervised classifier to effectively adapt itself after deployment from partial user feedback. the experiments show that our technique is successful, in that it improves over the initial supervised system. more specifically, in document classi-fication experiments, it matches an off-line supervised system trained with all the true labels, although it has only access to the binary feedback. more importantly, the experiments in two widely used cqa datasets, quac and doqa, confirm that it is feasible to improve a cqa system after deployment. in the doqa experiments, the cqa system is trained off-line in one domain (wikipedia) and then deployed in other domains, letting the users improve it via their partial feedback by interacting with the system. in this setting, the performance of our model also matches that of the fully supervised model which is fine-tuned with true labels rather than binary feedback. moreover, feedback-weighted learning is shown to be effective in two deep learning architectures, including a multi-layer feed forward network and a high-performing pre-trained transformer fine-tuned in the task. this work uses simulated feedback derived from gold standard labels. in the future we plan to modify feedback-weighted learning to cope with noisy feedback, as well as modifying it to work with fewer samples per query.