Continually Improving Extractive QA via Human Feedback (2023)

Abstract: We study continually improving an extractive question answering (QA) system via human user feedback. We design and deploy an iterative approach, where information-seeking users ask questions, receive model-predicted answers, and provide feedback. We conduct experiments involving thousands of user interactions under diverse setups to broaden the understanding of learning from feedback over time. Our experiments show effective improvement from user feedback of extractive QA models over time across different data regimes, including significant potential for domain adaptation.

## introduction the deployment of natural language processing (nlp) systems creates ample opportunities to learn from interaction with users, who can often provide feedback on the quality of the system output. such feedback can be more affordable and abundant than annotations provided by trained experts. it can also be obtained throughout the system's lifetime, opening up opportunities for continual learning, foot_0 where the system improves over time, and evolves as the world and user preferences change. the combination of human feedback and continual learning presents exciting prospects, but is relatively understudied, partially because of the challenges it poses. mainly because it requires deploying models with human users over many interactions for both development and evaluation, for example to study the interplay between the learning signal and model design, the impact of the initial model, and learning progression over time. focusing on extractive question answering (qa), we study iteratively improving an nlp system by learning from human user feedback over time. we figure 1 : illustration of our setup. we alternate between interaction and learning phases. at the interaction phase, given a user question and context paragraph, the model predicts the question is unanswerable or a span in the paragraph as the answer. the human user provides "correct", "partially correct" or "wrong" feedback to the model answer by validating the answer in context. the system aggregates user feedback, and the model learns from feedback data at the learning phase by updating its parameters, with the goal of improving over time. our setup is designed to study practical interaction with users. following prior work on collecting information-seeking queries @xcite , we prompt users to ask ques-tions that they do not know the answers to. this naturally elicits questions that cannot be answered under the fixed-evidence qa scenario @xcite . we observe that the presence of unanswerable questions dramatically impacts learning, making the process sensitive to the model's calibration on answerability. we address this by modeling question answerability separately from span extraction: our model first predicts if the question is answerable and only then extracts an answer span. we use a simple three-option feedback signal ("correct", "partially correct", or "wrong"). while less informative than more complex cues, such as natural language, such feedback imposes little interaction overhead. we map the feedback signal to manually-specified reward values, and formulate learning as a contextual bandit problem. we conduct multiple studies, focusing on low data regimes, with the aim of demonstrating robustness to challenging scenarios where only a small fraction of users provide feedback, and rapid improvement is necessary with little feedback. over nine rounds of deployment with over 1,800 user interactions, our approach shows an overall improvement from 39 to 67 f1 score on our newly collected wikipedia-based data. we conduct extensive studies of our approach, including showing effective domain adaptation from a different source domain that does not even support unanswerable questions. taken together, our results demonstrate the potential of nlp systems to learn from user feedback over time, specifically for extractive qa. our data and codebase are publicly available at https://github.com/lil-lab/qa-from-hf . ## related work our use of human feedback is related to work recently-termed reinforcement learning from human feedback (rlhf; e.g., @xcite @xcite @xcite . largely, these methods rely on soliciting pair-wise comparisons from annotators, which are used to train a reward model to be used in an rl process. we adopt a different approach: soliciting feedback from users on single outputs to their queries, and mapping the feedback to reward values to be used in offline contextual bandit learning. an important consideration motivating our choice is that pair-wise comparison, although suitable for paid annotators, is less suitable for soliciting feedback from actual users. head-to-head comparison between learning a reward model and directly mapping feedback to reward values, as we do, remains an important direction for future work. human feedback has also been studied without rl @xcite . a critical distinction of our work is the focus on continual learning (i.e., iteratively running our process over many rounds) and the dynamics it creates, whereas the work mentioned above (both using and not using rl) focused on improving models a single time. 2 learning from explicit or implicit feedback for language tasks was studied beyond recent interest in rlhf, including for machine translation @xcite , semantic parsing @xcite ), question answering @xcite , and chatbots @xcite . similar to the work mentioned earlier, this line of work did not explore iterative continual learning, as we emphasize. the iterative process was studied in the context of embodied instruction generation @xcite and following @xcite . in contrast, we study qa on a wide range of wikipedia articles, data with high lexical diversity. others obtained complete labels as feedback @xcite , a process with higher overhead. rl using human feedback has also been studied for non-language problems (e.g., @xcite @xcite . prior work on learning from interaction for qa used synthetic interactions and feedback simulated from supervised data @xcite . we study human feedback from information-seeking human users who frequently pose challenging questions which may be unanswerable from the given document. @xcite proposed a process that involves crowdworkers providing rating and explanation to given for given question-answer pairs to improve a qa model postdeployment. they control data quality with manual reviewing and multiple annotations. our setup has lower interaction overhead, at the cost of providing a less informative, and at times noisy signal. we also go beyond one-time improvement in studying iterative deployment, providing insight into how the model improves over time. ## interaction scenario we focus on the task of extractive qa, which creates an intuitive feedback solicitation scenario. it is relatively easy to visualize the model output (i.e., a span of text) in the context it is extracted from (i.e., the evidence paragraph), for the user to verify the answer. automated evaluation is also well understood @xcite , allowing reliable measurement of system performance over time. we deploy our qa system in rounds. each round starts with an interaction phase, followed by a learning phase. at the interaction phase, users interact with a fixed, deployed model and provide feedback. we aggregate this interaction data until we collect a fixed amount of feedback data to enter a learning phase. the feedback is collected during natural user interactions (i.e., the model is deployed to fulfil its task of answering user questions). at the learning phase, we update the model parameters based on the aggregated feedback data. because we observe no new feedback data during the learning phase, this creates an offline learning scenario @xcite ), 3 which is practical for deployed systems. except data aggregation, it requires no integration of the learning process into the deployed interactive system. the separation between deployment and training also enables sanity checks before deploying a new model, and for hyperparameter tuning as in supervised learning. each interaction starts with a user posing a question. the model computes an answer, and returns it to the user alongside a visualization of it in the context text from which it was extracted. the user provides feedback, by selecting one of three options: "correct", "partially correct" or "wrong". formally, let a question q be a sequence of m tokens ⟨q 1 , . . . , q m ⟩ and a context text c be a sequence of n tokens ⟨c 1 , . . . , c n ⟩. a qa model at round ρ parameterized by θ ρ computes two probability distributions: a binary distribution indicating if the question is answerable or not by the context text p u (u|q, c; θ ρ ), where @xmath0, unans}; and a distribution over answer spans in the context text p s (i, j|q, c; θ ρ ), where i, @xmath1, n] and @xmath2. if arg max @xmath3,unans} p u (u|q, c; θ ρ ) = unans, the model returns to the user that the question is unanswerable. otherwise, the model returns the highest probability span arg max i,j p s (i, j|q, c; θ ρ ). given 3 our overall iterative process can also be seen as batched contextual bandit @xcite . the model output, the user provides feedback @xmath4, partially-correct, wrong}. each user interaction generates a data tuple (q, c, û, î, ĵ, f, θ ρ ), where û is the binary answerability classification, and î and ĵ are the start and end indices of the extracted answer, if the question is classified as answerable. ## method we initialize the model with supervised data, and improve it by learning from user feedback through an offline contextual bandit learning process. ## experimental setup deployment setup we study our approach using amazon mechanical turk. we use wikipedia data, a relatively general domain, thereby making the data we observe lexically rich and topically diverse. our interaction scenario is designed to elicit information-seeking behavior. following prior work @xcite , users are not given the context text when they pose the question, and are instructed to ask questions they do not know the answer to. this results in a significant number of questions that cannot be answered given the evidence context. to increase user informationseeking engagement, we do not dictate the topic to ask about, but allow users to select a topic (e.g., "saladin") and an aspect (e.g., "death") from a set that is generated randomly for each interaction @xcite . 5 each topic-aspect pair is associated with an evidence paragraph, which serves as context to the model. we extract topics, aspects, and evidence paragraphs from wikipedia. 6 we focus on learning from natural user feedback, so we do not perform posthoc filtering of noise beyond removing adversarial crowdworkers (e.g., workers who always ask the same question). 7 appendix b provides more details about our interface and data collection. appendix a gives training implementation details (e.g., hyperparameters). evaluation data it is critical for test data to come from the distribution the users create while interacting with the system for the evaluation to reflect the system performance. this makes existing benchmarks of lesser relevance to our evaluation. we collect a held-out testing set that matches the observed distribution by interleaving the process in our deployment. we design a data annotation procedure based on our regular interaction task, and randomly opt to ask workers to provide an answer to the question they just posed rather than showing them the model answer and soliciting their feedback. this branching in the process happens only after they ask the question, and workers see no indication for it when posing the question, so the questions we collect this way follow the same distribution the systems see. we collect around 100 such annotated examples each round during each of our experiments. this is a small set, but unified across rounds within each experiment, it results in a set large enough for reliable testing. to increase the quality of the annotation, we separately collect two additional annotations for each such test question. appendix b provides further details on the test data collection and annotation. we also evaluate our approach on the english portion of tydiqa @xcite development set (954 examples, excluding yes/no questions), which was collected in a setting relatively similar to our evaluation data. each example includes a question, a context paragraph, and a set of three human reference annotations. evaluation metrics we focus on how different measures develop throughout the system's deploy6 our pool of topics is a subset of previously compiled popular entities @xcite based on the number of contributors and backlinks. aspects of each topic correspond to section titles on its wikipedia page. each topic in our pool contains at least four sections, with each section shorter than 490 tokens (excluding subsections). 7 qualitative analysis of 100 randomly sampled examples from the first round of our long-term study (section 6.1) shows a feedback noise rate of 18%. ment. we compute deployment statistics on the aggregated interactions, including answerability prediction, feedback, and the reward values it maps to. we use the testing data to compute various testing statistics on each deployed model, including token f1 and answerability classification accuracy. for both datasets, we follow prior work @xcite to consolidate the annotations. when the annotators disagree on the answerability of the question, we take the majority vote as the ground truth. we compute token-level f1 against the three annotations by considering the reference answer that will give the highest f1 score. ## results and analysis we conduct two deployment studies: a long-term deployment to study the effectiveness and dynamics of the process (section 6.1), and a shorterterm study to evaluate the impact of certain design choices (section 6.2). we also evaluate the difference between iterative rounds and a one-time improvement (section 6.3). section 6.4 provides an additional experiment analyzing the sensitivity of our learning process to the observed data. ## conclusion we deploy an extractive qa system, and improve it by learning from human user feedback, observing strong performance gains over time. our experiments show such continual improvement of systems is compelling for domain adaptation, providing a practical avenue for practitioners to bootstrap nlp systems from existing academic datasets, even when the domain gap is relatively large. we also study the effects of the update schedule (i.e., how frequently to train and re-deploy a new model) and model design (i.e., by using a separate answerability classification head). through our focus on a lexically-rich user-facing scenario, we hope our study will inspire further research of nlp problems in interactive scenarios and the learning potential user interactions provide. important directions for future work include learning from other, more complex signals (e.g., natural language feedback), studying non-extractive scenarios, and developing robustness to adversarial behaviors (e.g., attempts to sabotage the system through feedback).