Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users (2023)

Abstract: Recent work has shown that large-scale annotated datasets are essential for training state-of-the-art Question Answering (QA) models. Unfortunately, creating this data is expensive and requires a huge amount of annotation work. An alternative and cheaper source of supervision is given by feedback data collected from deployed QA systems. This data can be collected from tens of millions of user with no additional cost, for real-world QA services, e.g., Alexa, Google Home, and etc. The main drawback is the noise affecting feedback on individual examples. Recent literature on QA systems has shown the benefit of training models even with noisy feedback. However, these studies have multiple limitations: (i) they used uniform random noise to simulate feedback responses, which is typically an unrealistic approximation as noise follows specific patterns, depending on target examples and users; and (ii) they do not show how to aggregate feedback for improving training signals. In this paper, we first collect a large scale (16M) QA dataset with real feedback sampled from the QA traffic of a popular Virtual Assistant.Second, we use this data to develop two strategies for filtering unreliable users and thus de-noise feedback: (i) ranking users with an automatic classifier, and (ii) aggregating feedback over similar instances and comparing users between each other. Finally, we train QA models on our filtered feedback data, showing a significant improvement over the state of the art.

## introduction large pre-trained language models, e.g., based on the transformer neural network @xcite , have recently improved natural language processing and information retrieval in several tasks, e.g., document classification @xcite , question answering (qa) @xcite , neural retrieval @xcite . ## related work training qa systems with feedback data after models deployment received considerable attention in the past years. campos et al. (2020) for instance, simulated a scenario where an initial deployed machine reading model is continuously trained through feedback responses, showing promising sandbox performance. authors simulated feedbacks as positive whenever the answer span predicted by the system matches the gold span exactly, and negative otherwise. a similar scenario was considered by @xcite , where feedback was collected through crowdworkers. the authors themselves pointed out the limits of these studies as the results were based on artificial data distribution. other authors explored feedback such as explanation of incorrect responses by chatbots @xcite . however, the feedback in these studies is automatically generated using heuristics. similarly, @xcite collected human explanations for commonsense qa in the form of natural language sequences, and used the data to improve existing models with state-of-theart performance. the main weakness of previous work is that most of existing analyses (i) are based on artificially generated data @xcite @xcite or (ii) use crowdsourced workload (i.e., annotators) to simulate real feedback data. as a consequence, these approaches are not suitable for industrial scenarios since they do not consider noise distribution of real users' generated data. in contrast, our work is based on real user data from a virtual assistant, which provides answer using a web-based qa system. our results are generalizable to most industrial scenarios targeting open domain qa. ## operational setting we consider the task of selecting the correct answer sentence among a set of candidates extracted from web-documents retrieved for a given question. formally, let σ * be the set of strings (or general sentences) and q ⊆ σ * be the set of questions according to a certain input distribution. given an input question @xmath0.g., returned by a search engine), the answer selector can be defined as a function r : q × σ * → r, which assigns a probability score to each q/a pair, r(q, s i ), and returns the answer associated with the highest ranked pair, i.e., arg max @xmath1...k r(q, s i ). we use the state-of-the-art model for answer sentence selection @xcite , which implements r with a transformer model. this encodes an input pair (q, s i ) as [cls] q [sep] s i [eos] and returns the associated score through a classification head on top of the [cls] token. ## training with de-noised feedback the standard approach to de-noise training examples is to assign them different weights according to their reliability. finding these weights for individual feedback instances is an open problem. we propose two methods: the first is based on a completely new idea (to our knowledge): we observed that different users have different accuracy in assessing the correctness of answers. since manually labeling millions of users is not feasible, we use an automatic answer selector classifier, trained on off-the-shelf data. the second approach is based on standard collaborative filtering applied to user clusters, which are created by comparing questions using state-of-the-art text similarity techniques. ## empirical assessment we divided our experiments in 3 groups: first, we analyze the quality of our filtered data through manual evaluation. then, we show that the massive amount of noisy feedback can improve the performance of qa models already trained on large-scale high-quality annotated data. finally, we analyze the impact of the de-noising strategies described in the previous sections. ## conclusion feedback data represents a huge and convenient source of training data, which can be used to im-prove the performance of deployed qa systems. however, the noise affecting feedback can degrade model performance. this paper introduces two ml approaches to filter feedback data and to reduce the amount of noise. our filters are based on the assumption that users act differently from each other. thus, their behaviour induces different reliability, which if modeled correctly can help to build more effective training data. we used a large set of question, answer, and feedback tuples (16m) sampled from a commercial virtual assistant to validate this hypothesis. our extensive empirical assessment clearly shows that filtered feedback can significantly improve the performance of a deployed qa system, even when the models are trained on massive high-quality annotated resources. note that this work does not aim to compare different filtering methods to elect a superior approach. we conjecture that the collaborative filtering can be further improved, for instance by deeply analyzing different clustering approaches or embeddings extractors. on the contrary, our goal is (i) to highlight the importance and impact of using real feedback data to improve the performance of industrial qa models, and (ii) to provide insights for future research directions. to the best of our knowledge, this work represents the first analysis on real feedback data and its integration into model training. these findings reveal promising directions to improve deployed qa systems.