Synthetic Question Value Estimation for Domain Adaptation of Question Answering (2022)

Abstract: Synthesizing QA pairs with a question generator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models. Since synthetic questions are often noisy in practice, existing work adapts scores from a pretrained QA (or QG) model as criteria to select high-quality questions. However, these scores do not directly serve the ultimate goal of improving QA performance on the target domain. In this paper, we introduce a novel idea of training a question value estimator (QVE) that directly estimates the usefulness of synthetic questions for improving the target-domain QA performance. By conducting comprehensive experiments, we show that the synthetic questions selected by QVE can help achieve better target-domain QA performance, in comparison with existing techniques. We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable performance to the fully-supervised baselines.

## introduction question answering (qa) systems based on pretrained language models such as bert @xcite have recently achieved promising performance in machine reading comprehension. however, neural qa systems trained on one domain may not generalize well to another, leaving it challenging to deploy such systems on new domains that lack large-scale qa training data 2 . in this paper, we are interested in semi-supervised domain adaptation: we aim to build a target qa model with source-domain data and a small number of target-domain annotated qa pairs. figure 1 : existing work repurposes a pretrained qa (or qg) model to evaluate the quality of the generated questions, which is not directly associated with the target-domain qa performance and may select questions that are semantically-mismatched or ask about a simple fact. in contrast, our question value estimator (qve) learns to select useful questions with targetdomain qa performance gain as direct feedback. due to high annotation costs, existing work @xcite @xcite @xcite proposes to synthesize target-domain qa pairs via neural question generation (qg) models. the synthetic data are then used to train a qa model on the target domain. in practice, however, the generated questions are often of low quality, such as being semantically mismatched with their paired answers or asking about simple facts (figure 1 ). including all such questions for qa training is less likely to bring substantial improvements. this inspires us to study a crucial problem: given a set of target-domain synthetic qa pairs, how to select high-quality ones that are useful to improve target-domain qa training? to address the problem, @xcite propose the roundtrip consistency (rtc) method, which filters foot_1 questions that cannot be correctly answered by a pretrained qa model. other work @xcite considers using the generation log likelihood by the qg model (lm score) as a metric to filter noisy questions (figure 1 , top). although these filtering techniques have been shown to improve the question quality to some extent @xcite , they are not directly optimized for selecting questions that can improve qa performance on the target domain. for example, some useful but difficult questions (e.g., the last example in figure 1 ) may be filtered by the roundtrip method, since they cannot be answered correctly by the pretrained qa model. however, these questions are often crucial to further improving qa performance when added into training. in this paper, we propose a question value estimator (qve) (figure 1 , middle) to select questions that can improve qa performance on the target domain. qve takes in generated qa examples and outputs real-valued scores (i.e., question values), which are expected to represent the usefulness of generated questions in terms of improving targetdomain qa performance. however, training the qve model towards this goal is challenging due to the lack of supervision (i.e., true question values). to solve the problem, we propose to train the qve with direct qa feedback from the target domain. intuitively, if a batch of synthetic questions (when used for training) leads to increasing accuracy of the target-domain qa model, qve should assign high values to them; the more the accuracy increases, the higher the question values should be. thus, we optimize qve with the target-domain qa performance gain after adding the selected questions into training. more formally, given the discrete and non-differentiable question selection process, we formulate the question selection of qve as a reinforcement learning @xcite problem (figure 2 ). the qve receives a batch of synthetic samples each time and learns to select highquality ones based on their estimated values. the selected samples are then used to train the targetdomain qa model, with the resulting performance gain (on the available target-domain annotations) as the reward. the reward guides the optimization of qve such that it will eventually make proper question value estimation and selection. to evaluate the qve model, we instantiate the qg and the qa model based on the pretrained bart @xcite and bert @xcite , respectively. by carrying out comprehensive experiments on four commonly-used reading comprehension datasets @xcite @xcite , we show that: (1) our qve model trained with the target-domain qa feedback substantially outperforms the question selection techniques trained without direct qa feedback @xcite . (2) when using our qve model to select synthetic questions, qa models can achieve comparable performance to fully-supervised baselines while using only 15% of the full target-domain annotations, which indicates that our method can greatly alleviate human annotation effort in practice. (3) to understand why qve brings superior improvement, we conduct human evaluation and find that qve can better identify semantically-matched and difficult questions. the task is to build a qa model that can accurately answer questions on the target domain, given d s , d t , and c t . ## question value estimator (qve) formally, we design a question value estimator (qve), e γ , which takes in a synthetic qa example (c l , q l , a l ) (for simplicity, we omit the superscript t) and outputs a score indicating its "value," i.e., v @xmath0, q l , a l ). the "value" can imply "the potential for improving the target-domain qa performance when being used as a training sample". with this score, one can select most useful synthetic examples for the target-domain qa training. we use a bert model as the backbone of the qve. specifically, we concatenate the context, question and answer as input to the qve, and use bert to encode the sequence @xcite . in our preliminary experiments, we find that adding the answer (start index and end index) probabilities (p s , p e ) by a pretrained qa model as additional features to the hidden representation h can accelerate the qve training convergence and lead to better performance. thus, we add these two features (p s , p e ) followed by linear transformations of the original hidden representation, and then build a linear classifier to output the question value. learning such a question value estimator is challenging because we do not have direct supervision on the true value or usefulness of a synthetic question. we discuss two straightforward baselines to train qve in section 4.1, and a more advanced one based on reinforcement learning in section 4.2. ## conclusion we propose a question value estimator to estimate the usefulness of synthetic questions and select useful ones for improving target-domain qa train7 we treat it as a binary classification problem here: if a question is selected, the prediction is 1; 0 otherwise.