Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering (2021)

Abstract: In this paper we propose a novel approach towards improving the efficiency of Question Answering (QA) systems by filtering out questions that will not be answered by them. This is based on an interesting new finding: the answer confidence scores of state-of-the-art QA systems can be approximated well by models solely using the input question text. This enables preemptive filtering of questions that are not answered by the system due to their answer confidence scores being lower than the system threshold. Specifically, we learn Transformer-based question models by distilling Transformer-based answering models. Our experiments on three popular QA datasets and one industrial QA benchmark demonstrate the ability of our question models to approximate the Precision/Recall curves of the target QA system well. These question models, when used as filters, can effectively trade off lower computation cost of QA systems for lower Recall, e.g., reducing computation by ~60%, while only losing ~3-4% of Recall.

## introduction question answering (qa) technology is at the core of several commercial applications, e.g., virtual assistants such as alexa, google home and siri, serving millions of users. optimizing the efficiency of such systems is vital to reduce their operational costs. recently, there has been a large body of research devoted towards reducing the compute complexity of retrieval @xcite and transformer-based qa models @xcite ). an alternate solution for improving qa system efficiency aims to discard questions that will most probably be incorrectly answered by the system, using automatic classifiers. for example, @xcite ; @xcite aim to capture the grammaticality and well-formedness of questions. however, these methods do not take the specific answering ability of the target system into account. in practice, qa systems typically do not answer a significant portion of user questions since their answer scores could be lower than a confidence threshold @xcite , tuned by the system to achieve the required precision. for example, qa systems for medical domains exhibit a high precision since providing incorrect/imprecise answers can have critical consequences for the end user. based on the above rationale, discarding questions that will not be answered by the qa system presents a remarkable cost-saving opportunity. however, applying this idea may appear unrealistic from the outset since the qa system must first be executed to generate the answer confidence score. in this paper, we take a new perspective on improving qa system efficiency by preemptively filtering out questions that will not be answered by the system, by means of a question filtering model. this is based on our interesting finding that the answer confidence score of a qa system can be well approximated solely using the question text. our empirical study is supported by several observations and intuitions. first, the final answer confidence score from a qa system (irrespective of its complex pipeline) is often generated by a transformer-based model. this is because transformer-based models are used for answer extraction in most research areas in qa with unstructured text, e.g., machine reading(mr) @xcite , and answer sentence selection(as2) @xcite . second, more linguistically complex questions have a lower probability to be answered. language complexity correlates with syntactic, semantic and lexical properties, which have been shown to be well captured by pre-trained language models (lms) @xcite . thus, the final answer extractor will be affected by said complexity, suggesting that we can predict which questions are likely to be unanswered just using their surface forms. third, pre-training transformer-based lms on huge amounts of web data enables them to implicitly capture the frequency/popularity of general phrases foot_0 , among which entities and concepts play a crucial role for answerability of questions. thus, the contextual embedding of a question from a transformer lm is, to some extent, aware of the popularity of entities and concepts in the question, which impacts the retrieval quality of good answer candidates. this means that a portion of the retrieval complexity of a qa system can also be estimated just using the question. most importantly, we only try to estimate the answer score from a qa system and not whether the answer provided by the system for a question is correct or incorrect (the latter being a much more difficult task). following the above intuitions, we distill the knowledge of qa models, using them as teachers, into transformer-based models (students) that only operate on the question. once trained, the student question model can be used to preemptively filter out questions whose answer score will not clear the system threshold, translating to a proportional reduction in the runtime cost of the system. more specifically, we propose two loss objectives for training two variants of this question filter: one with a regression head and one with a classification head. the former attempts to directly predict the continuous score provided by the qa system. the latter aims at learning to predict if a question will generate a @xmath0, which is the answer confidence threshold the qa system was tuned to. we perform empirical evaluation for (i) showing the ability of our question models to estimate the qa system score; and (ii) testing the cost savings produced by our question filters, trading off with a drop in recall. we test our models on two qa tasks with unstructured text, mr and as2, using (a) three academic datasets: wikiqa, asnq, and squad 1.1; (b) a large scale industrial benchmark, and (c) a variety of different transformer architectures such as bert, roberta and elec-tra. specifically for (i), we compare the precision(pr)/recall(re) curves of the original and the new qa system, where the latter uses the question model score to trade-off precision for recall. for (ii), we show the cost savings produced by our question filters, when operating the original qa system at different precision values. the results show that: answer verification qa systems sometimes use an answer validation component in addition to the system threshold, which analyzes the answer produced by the system and decides whether to answer or abstain. these systems often use external entity knowledge @xcite @xcite for basing their decision to verify the correctness of the answer. recently @xcite propose to add a new mr model to reflect on the predictions of the original mr model to decide whether the produced answer is correct or not. other efforts @xcite @xcite have trained calibrators for verifying if the question should be answered or not. all these works are fundamentally different from our question filtering approach since they operate jointly on the question and generated answer, thereby requiring the entire computation to be performed by the qa system before making a decision. our work operates only on the question text to preemptively decide whether to filter it or not. thus the primary goal of these existing works is to improve the precision of the answering model by not answering when not confident, while our work aims to improve efficiency of the qa system and save runtime compute cost. query performance prediction pre-retrieval query difficulty prediction has been previously explored in information retrieval @xcite . previous works @xcite @xcite @xcite target p(a|q, f ), ground truth probability of an answer a to be correct, given a question q and a feature set f in input using simple linguistic (e.g., parse trees, polysemy value) and statistical (e.g., query term statistics, pmi) methods; while we target the qa-system score s(a|q, f ), the probability of an answer to be correct as estimated by the system. this task is more semantically driven than syntactically, and enables the use of large amounts of training data without human labels of answer correctness. efficient qa several works on improving the efficiency of the retrieval involve using a cascade of rerankers to quickly identify good documents @xcite @xcite , nonmetric matching functions for efficient search @xcite , etc. towards reducing compute of the answer model, the following techniques have been explored: multi-stage ranking using progressively larger models @xcite , using intermediate representations for early elimination of negative candidates @xcite , combining separate encoding of question and answer with shallow dnns @xcite , and the most popular being knowledge distillation @xcite to train smaller transformer models with low inference latencies (distilbert @xcite , tinybert @xcite , mobilebert @xcite , etc.) ## preliminaries and problem setting we first provide details of qa systems and explain the cost-saving opportunity space when they operate at a given precision (or answer score threshold). ## modelling qa system scores using input questions we propose to use a distillation approach to learn a model operating on questions that can predict the confidence score of the qa system, within a certain error bound. we denote the qa system with ω(r, s,m τ ), and the question model by f (as we will use it to filter out questions preemptively). intuitively, f aims at learning how confident the answer model m is on answering a particular question when presented with a set of candidate answers from a retrieval system (r, s). for a question q, we indicate the set of answer the output from m for q and s corresponds to the score for the best candidate/span: m(q, s) = max @xmath1, s). we train a filter f m for m using a regression head to directly predict the score of m irrespective of the threshold τ using the loss: the novelty of our proposed approach from standard distillation techniques @xcite @xcite stems from the fact that, unlike the standard setting, in our case the teacher m and the student f operate on different inputs: f only on the questions while m on question-answer pairs. this makes our task much more challenging as f needs to approximate the probability of m fitting all answer candidates for a question. since our f does not predict if an answer provided by m is correct/incorrect, we don't require labels of the qa dataset for training f (f's output only depends on predictions of m). this enables large scale training of f without any human supervision using the predictions of a system ω(r, s,m τ ) and a large number of questions. to use the trained f for preemptively filtering out questions, we use a threshold on the score of f. henceforth, we refer to the threshold of m by τ 1 and that of f by τ 2 . any question q for which f(q)≤τ 2 , gets filtered out. using the question filter, we define the new qa system as ω(f τ 2 , r, s, m τ 1 ) where the filter f can be trained using eq. 1 or 2 (f m or f m τ 1 ). ## experiments first, we compare how well our models f can approximate the answer score of m. then we optimize τ 1 and τ 2 on the dev. set to precisely estimate the cost savings that we can obtain with the application of our approach to questions filtering. we also compare it with different baseline models for f from previous works on question filtering. ## conclusion and future work in this paper, we have presented a novel paradigm of training a question filter to capture the semantics of a qa system's answering capability by distilling the knowledge of the answer scores from it. our experiments on three academic and one industrial qa benchmark show that the trained question models can estimate the pr/re curves of the qa system well, and can be used to effectively filter questions while only incurring a small drop in recall. an interesting future work direction is to analyze the impact/behavior of the question filters in a cross-domain setting, where the training and testing corpora are from different domains. this would allow examining the transferability of the semantics learned by the question filters. a complementary future work direction could be knowledge distillation from a sophisticated answer verification module like @xcite @xcite . in addition to providing efficiency gains, the question filters could be used to qualitatively study the characteristics of questions that are likely to lead to low answer confidence scores. this can (i) help error analysis for improving the accuracy of qa systems, and (ii) be used for efficient sampling of training questions that are harder to be answered by the target qa system. our approach for training the question filters proposes the idea of partial-input knowledge distillation (e.g., using only questions instead of qa pairs). this concept can possibly be extended to other nlp problems for achieving compute efficiency gains, improved explainability (e.g., to what extent a partial-input influences model prediction) and qualitative analysis.