Simulating Bandit Learning from User Feedback for Extractive Question Answering (2022)

Abstract: We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on few examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation effort, but instead improving the system on-the-fly via user feedback.

## introduction explicit feedback from users of nlp systems can be used to continually improve system performance. for example, a user posing a question to a question-answering (qa) system can mark if a predicted phrase is a valid answer given the context from which it was extracted. however, the dominant paradigm in nlp separates model training from deployment, leaving models static following learning and throughout interaction with users. this approach misses opportunities for learning during system usage, which beside several exceptions we discuss in section 8 is understudied in nlp. in this paper, we study the potential of learning from explicit user feedback for extractive qa through simulation studies. extractive qa is a popular testbed for language reasoning, with rich prior work on datasets (e.g., @xcite , task design @xcite , and model architecture development @xcite . learning from interaction with users remains relatively understudied, even though qa is well positioned to elicit user feedback. an extracted answer can be clearly visualized within its supporting context, and a language-proficient user can then easily validate figure 1 : illustration of an interaction setup for learning from user feedback for qa, and its potential. given a user question, the system outputs an answer and highlights it in its context. the user validates the answer given the context with binary feedback. we show performance progression from one of our online learning experiments on squad with hand-crafted illustrative examples at two time steps. if the answer is supported or not. 1 this allows for simple binary feedback, and creates a contextual bandit learning scenario @xcite . figure 1 illustrates this learning signal and its potential. we simulate user feedback using several widely used qa datasets, and use it as a bandit signal for learning. we study the empirical characteristics of the learning process, including its performance, sensitivity to initial system performance, and tradeoffs between online and offline learning. we also simulate zero-annotation domain adaptation, where we deploy a qa system trained from supervised data in one domain and adapt it solely from user feedback in a new domain. this learning scenario can mitigate fundamental problems in extractive qa. it reduces data collection costs, by delegating much of the learning to interaction with users. it can avoid data collection artifacts because the data comes from the actual system deployment, unlike data from an annotation effort that often involves design decisions immaterial to the system's use case. for example, sharing question-and answer-annotator roles @xcite , which is detrimental to emulate information seeking behavior @xcite . finally, it gives systems the potential to evolve over time as the world changes @xcite . our simulation experiments show that user feedback is an effective signal to continually improve qa systems across multiple benchmarks. for example, an initial system trained with a small amount of squad @xcite annotations (64 examples) improves from 18 to 81.6 f1 score, and adapting a searchqa @xcite system to squad through user feedback improves it from 45 to 84 f1 score. our study shows the impact of initial system performance, trade-offs between online and offline learning, and the impact of source domain on adaptation. these results create the base for future work that goes beyond simulation to use feedback from human users to improve extractive qa systems. our code is publicly available at https://github.com/ lil-lab/bandit-qa . ## learning and interaction scenario we study a scenario where a qa model learns from explicit user feedback. we formulate learning as a contextual bandit problem. the input to the learner is a question-context pair, where the context paragraph contains the answer to the question. the output is a single span in the context paragraph that is the answer to the question. given a question-context pair, the model predicts an answer span. the user then provides feedback about the model's predicted answer, which is used to update the model parameters. we intentionally experiment with simple binary feedback and basic learning algorithms, to provide a baseline for what more advanced methods could achieve with as few assumptions as possible. background: contextual bandit learning in a stochastic (i.i.d.) contextual bandit learning problem, at each time step t, the learner independently observes a context foot_1 x (t) ∼ d sampled from the data distribution d, chooses an action y (t) according to a policy π, and observes a reward r (t) ∈ r. the learner only observes the reward r (t) corresponding to the chosen action y (t) . the learner aims to minimize the cumulative regret. intuitively, regret is the deficit suffered by the learner relative to the optimal policy up to a specific time step. formally, the cumulative regret at time t is computed with respect to the optimal policy π * ∈ arg max π∈π e (x,y,r)∼(d,π) [r]: scenario formulation let a question q be a sequence of m tokens q 1 , . . . , q m and a context paragraph c be a sequence of n tokens c 1 , . . . , c n . an extractive qa model foot_3 π predicts a span ŷ = c i , . . . , c j where i, @xmath0, n] and @xmath1. when relevant, we denote π θ as a qa model parameterized by θ. we formalize learning as a contextual bandit process: at each time step t, the model is given a question-context pair (q (t) , c(t) ), predicts an answer span ŷ, and receives a reward r (t) ∈ ir. the learner's goal is to maximize the total reward t @xmath2. this formulation reflects a setup where, given a question-context pair, the qa system interacts with a user, who validates the model-predicted answer in context, and provides feedback which is mapped to numerical reward. algorithm 1 online learning. receive a question q(t) and context c(k) 3: predict an answer ŷ(t) ← arg maxy π θ (y | q(t) , c(t) ) 4: observe a reward r (t) 5: update the model parameters θ using the gradient r (t) ∇ θ log π θ (ŷ (t) | q(t) , c(t) ) 6: end for learning algorithm we learn using policy gradient. our learner is similar to @xcite ), but we use arg max to predict answers instead of monte carlo sampling from the model's output distribution. 5 we study online and offline learning, also referred to as on-and off-policy. in online learning (algorithm 1), the model identity is maintained between prediction and update; the parameter values that are updated are the same that were used to generate the output receiving reward. this entails that a reward is only used once, to update the model after observing it. in offline learning (algorithm 2), this relation between update and prediction does not hold. the learner observes reward, often across many examples, and may use it to update the model many times, even after the parameters drifted arbitrarily far from these that generated the prediction. in practice, we observe reward for the entire length of the simulation (t steps) and then update for e epochs. the reward is re-weighted to provide an unbiased estimation using inverse propensity score @xcite . we clip the debiasing coefficient to avoid amplifying examples with large coefficients (line 10, algorithm 2). in general, offline learning is easier to implement because updating the model is not integrated with its deployment. offline learning also uses a training loop that is similar to optimization practices in supervised learning. this allows to iterate over the data multiple times, albeit with the same feedback signal on each example. however, online learning often has lower regret as the model is updated after each interaction. it may also lead to higher overall performance, because as the model improves early on, it may observe more positive feedback overall, which is generally more informative. we empiri5 early experiments showed that sampling is not as beneficial as arg max, potentially because of the relatively large output space of extractive qa. @xcite made a similar observation for semantic parsing, and @xcite used arg max predictions for bandit learning in statistical machine translation. algorithm 2 offline learning. receive a question q(t) and context c(t) 3: predict an answer ŷ(t) ← arg maxy π θ (y | q(t) , c(t) ) 4: observe a reward r (t) 6: end for 7: for e epochs do 8: compute clipped importance-weighted reward according to the current model parameters: evaluating performance we evaluate model performance using token-level f1 on a held-out test set, as commonly done in the qa literature @xcite . we also estimate the learner regret (equation 1 ). computing regret requires access to the an oracle π * . we use human annotation as an estimate (section 3). 6 comparison to supervised learning in supervised learning, the data distribution is not dependent on the model, but on a fixed training set {(q (t) , c(t) , y (t) )} t @xmath3. in contrast, bandit learners are provided with reward data that depends on the model itself: {(q (t) , c(t) , ŷ(t) , r (t) )} t @xmath4, c(t) ) at time step t. such feedback can be freely gathered from users interacting with the model, while building supervised datasets requires costly annotation. this learning signal can also reflect changing task properties (e.g., world changes) to allow systems to adapt, and its origin in the deployed system use makes it more robust to biases introduced during annotation. ## simulation setup we initialize our model with supervised data, and then simulate bandit feedback using supervised data annotations. initialization is critical so the model does not return random answers, which are likely to be all bad because of the large output space. we use relatively little supervised data from the same domain for in-domain experiments (section 5 and 6) to focus on the data annotation re-duction potential of user feedback. for domain adaptation, we assume access to a large amount of training data in the source domain, and no annotated data in the target domain (section 7). reward we use supervised data annotations to simulate the reward. if the predicted answer span is an exact match index-wise to the annotated span, the learner observes a positive reward of 1.0, and a negative reward of -0.1 otherwise. 7 this reward signal is stricter than qa evaluation metrics (tokenlevel f1 or exact match after normalization). 8 noise simulation we study robustness by simulating noisy feedback via reward perturbation: randomly flipping the binary reward with a fixed probability of 8% or 20% as the noise ratio. foot_7 ## online learning we simulate a scenario where only a limited amount of supervised data is available, and the model mainly learns from explicit user feedback on predicted answers. we use 64, 256, or 1,024 in-domain annotated examples to train an initial model. this section focuses on online learning, where the learner updates the model parameters after each feedback is observed (algorithm 1). figure 2 presents the performance of in-domain simulation with online learning. the performance pattern varies across different datasets. bandit learning consistently improves performance on squad, hotpotqa, and nq across different amounts of supervised data used to train the initial model. the performance gain is larger with weaker initial models (i.e., trained on 64 supervised examples): 63.6 on squad, 42.7 on hotpotqa, and 40.0 on nq. bandit learning is not always effective on newsqa, triviaqa, and searchqa, especially with weaker initial models. this may be attributed to the quality of training set annotations, which determines the accuracy of reward in our setup. searchqa and triviaqa use distant supervision to match questions and relevant contexts from the web, likely decreasing reward quality in our setup. while newsqa is crowdsourced, @xcite report relatively low human performance (69.4 f1), possibly indicating data challenges that also decrease our reward quality. learning progres- feedback noise simulation figure 3 shows learning curves with simulated noise via different amounts of feedback perturbation (0%, 8%, or 20%). when perturbation-free simulation is effective, models remain robust to noise: 8% noise results in small fluctuations of the learning curve, but the final performance degrades minimally. starting with weaker initial models and learning with a higher noise ratio may cause learning to fail (e.g., simulation on squad with 64 initial examples and 20% noise). when online perturbation-free simulation fails, online learning with noisy feedback fails too. ## offline learning we simulate offline bandit learning (algorithm 2), where feedback is collected all at once with the initial model. the learning scenario follows the previous section: only a limited amount of supervised data is available (64, 256, or 1,024 in-domain examples) to train initial models. online vs. offline regret ## domain adaptation figure 4 shows online domain adaptation performance. on 22/30 configurations, online adaptation introduces significant performance gains (>2 f1 score). for example, adapting from triviaqa and searchqa to the other four domains improves performance by 27-72.8 f1. on hotpotqa, the model initially trained on triviaqa shows an impressive adaptation, improving from 0.2 f1 to 73 f1. 12 our simulations show reduced effectiveness when the target domain is either triviaqa or searchqa, likely because the simulated feedback is based on noisy distantly supervised data. for searchqa, the low performance of initial models from other domains may also contribute to the adaptation failure. as expected, this indicates the effectiveness of the process depends on the relation figure 5 shows learning curves for our simulation experiments. generally, we observe the choice of source and target domains influences adaptation rates. models quickly adapt to squad, hot-potqa, and nq, reaching near final performance with a quarter of the total feedback provided. on newsqa, models initially trained on triviaqa and searchqa adapt slower than those initially trained on other three datasets. on triviaqa, we observe little change in performance throughout simulation. on searchqa, only the model initially trained on triviaqa shows a performance gain. both searchqa and triviaqa include context paragraphs from the web, potentially making domain adaptation from one to the other easier. lastly, we compare bandit learning with initial models trained on a small amount of in-domain data (section 5) and initial models trained on a large amount of out-of-domain data. ## related work bandit learning has been applied to a variety of nlp problems including neural machine translation (nmt; @xcite @xcite , structured prediction @xcite , semantic parsing @xcite , intent recognition @xcite , and summarization @xcite . explicit human feedback has been studied as a direct learning signal for nmt @xcite , semantic parsing @xcite , and summarization @xcite . @xcite simulates bandit feedback to improve an mt system fully trained on a large annotated dataset, including analyzing robustness to feedback perturbations. our work shows that simulated bandit feedback is an effective learning signal for extractive question answering tasks. our work differs in focus on reducing annotation costs by relying on few annotated examples only to train the initial model, or by eliminating the need for in-domain annotation completely by relying on data in other domains to train initial models. implicit human feedback, where feedback is derived from human behavior rather than explicitly requested, has also been studied, including for dialogue @xcite and instruction generation @xcite . we focus on explicit feedback, but implicit signals also hold promise to improve qa systems. alternative forms of supervision for qa have been explored in prior work, such as explicitly providing fine-grained information @xcite . @xcite resembles our setting in seeking binary feedback to replace span annotation, but their goal is to create supervised data more economically. campos et al. (2020) proposes feedback-weighted learning to improves conversational qa using simulated binary feedback. their approach relies on multiple samples (i.e., feedback signals) per example, training for multiple epochs online by re-visiting the same questions repeatedly, and tuning two additional hyperparameters. in contrast, we study improving qa systems via feedback as a bandit learning problem. in both online and offline setups, we assume only one feedback sample per example. we also provide extensive sensitivity studies to the amount of annotations available, different model initialization, and noisy feedback across various datasets. domain adaptation for qa has been widely studied @xcite , including using data augmentation @xcite , adversarial training @xcite , contrastive method @xcite , back-training @xcite , and exploiting small lottery subnetworks @xcite . ## conclusion we present a simulation study of learning from user feedback for extractive qa. we formulate the problem as contextual bandit learning. we conduct experiments to show the effectiveness of such feedback, the robustness to feedback noise, the impact of initial model performance, the trade-offs between online and offline learning, and the potential for domain adaptation. our study design emphasizes the potential for reducing annotation costs by annotating few examples or by utilizing existing datasets for new domains. we intentionally adopt a basic setup, including a simple binary reward and vanilla learning algorithms, to illustrate what can be achieved with a relatively simple variant of the contextual bandit learning scenario. our results already indicate the strong potential of learning from feedback, which more advanced methods are likely to further improve. for example, the balance between online and offline learning can be further explored using proximal policy optimization (ppo; @xcite or replay memory @xcite . with welldesigned interface, human users may be able to provide more sophisticated feedback @xcite , which will provide a stronger signal compared to our binary reward. our aim in this study is to lay the foundation for future work, by formalizing the setup and showing its potential. this is a critical step in enabling future research, especially going beyond simulation to study using real human feedback for qa systems. another important direction for future work is studying user feedback for qa systems that do both context retrieval and answer generation @xcite , where assigning the feedback to the appropriate stage in the process poses a challenge. beyond extractive qa, we hope our work will inspire research of user feedback as a signal to improve other types of nlp systems.