Unsupervised Adaptation of Question Answering Systems via Generative Self-training (2020)

Abstract: BERT-era question answering systems have recently achieved impressive performance on several question-answering (QA) tasks. These systems are based on representations that have been pre-trained on self-supervised tasks such as word masking and sentence entailment, using massive amounts of data. Nevertheless, additional pre-training closer to the end-task, such as training on synthetic QA pairs, has been shown to improve performance. While recent work has considered augmenting labelled data and leveraging large unlabelled datasets to generate synthetic QA data, directly adapting to target data has received little attention. In this paper we investigate the iterative generation of synthetic QA pairs as a way to realize unsupervised self adaptation. Motivated by the success of the roundtrip consistency method for filtering generated QA pairs, we present iterative generalizations of the approach, which maximize an approximation of a lower bound on the probability of the adaptation data. By adapting on synthetic QA pairs generated on the target data, our method is able to improve QA systems significantly, using an order of magnitude less synthetic data and training computation than existing augmentation approaches.

## introduction supervised self-training methods have transformed applied machine learning recently. such tasks serve as "pre-training" for related downstream tasks, and have proven to be essential to attaining state-of-theart performance, particularly in nlp. bert-era transformer-based question answering systems have recently achieved impressive performance on several question-answering (qa) tasks. these systems are based on representations that have been pre-trained on self-supervised tasks such as word masking and sentence entailment, using massive amounts of data @xcite @xcite @xcite . nevertheless, additional pretraining closer to the end-task, such as training on synthetic qa pairs, has been shown to improve performance @xcite . while recent work has considered augmenting labelled data and leveraging large unlabelled datasets to generate synthetic qa data, directly adapting to target data has, to our knowledge, not been investigated in the context of bert-era modeling and performance levels. recently, roundtrip consistency (rtc) was introduced as a criteria for filtering synthetic questionanswer pairs on unlabelled data, and has demonstrated solid gains when applied to large unlabelled datasets to generate millions of rtc-validated pairs as task-specific pre-training data @xcite . such unsupervised self-training can be an effective way to de-emphasize low confidence predictions, adapt to the target input distribution, distill decoding procedures, and instill input response invariances. in this paper we present new theoretical justification for rtc, and explore novel iterative generalizations of rtc for adapting in a task-specific, target data specific manner. we show that most of these approaches optimize an approximation of a lower bound of the probability of the data, and thereby can, beyond self-training, potentially also adapt to explain the target data more effectively. under the formulation, the question-answering system is used as a surrogate likelihood function for the question and answer generators. in this manner, the difficult task of modeling the generation of entire contexts is avoided: instead abstractive parts of the context (the answers), whose locations are latent and estimated, are utilized, similar to an autoencoder, but with questions as the latent code, and latent answer inputs. by adapting on synthetic qa pairs generated on the target data, our method is able to improve qa systems significantly, using an order of magnitude less synthetic data and training computation than existing augmentation approaches. the main contributions of this paper are as follows: typically question-answering (qa) systems are trained to produce an abstractive answer a, given a question q, and the relevant local context c. the qa model is generally trained to maximize the probability of qa pairs in context: where {q t , a t , c t } denotes a set of humanannotated question-answer-context triples @xcite . supervised training of question-answering systems is effective, but ground-truth (gt) qa pairs are cumbersome and expensive to annotate. bootstrapping from pretrained representations such as bert significantly reduces the number of gt qa pairs that are required to train a high quality qa system, but current systems still currently require hundreds of thousands of gt qa pairs on data matching the style and content of the target data to perform at state-of-the-art levels. to augment an existing qa training set, several authors have used the supervised data to train question generators @xcite @xcite . as with the qa counterparts, these models are typically trained in a supervised manner: and then employed to augment the training set with additional, synthetic questions, for each groundtruth answer. most techniques that operate fully unsupervised also train an answer prior in the same manner, which generates answers for a given context, which the question generator then conditions on to generate questions @xcite : most existing methods for synthetic data augmentation utilize qa generators trained only on supervised data. this synthetic data is used either as pre-training data for the qa system, which is then fine-tuned on ground-truth data @xcite , or just simply as additional gt data for training. in this paper we investigate iteratively pretraining both the qa system and the qa generators on the synthetic data generated by the most recent fine-tuned qa generators. the iterative procedure is as follows: ## fundamentals of unsupervised self-training training a model on it's own predictions, or pseudotruth, has a long history in machine learning and speech and language applications @xcite . it generally leads to performance improvements, but why? the expected gradient of the log probability of a model p(y|x) trained on its own predictions is zero: e y∼p θ (y|x) [∇ θ log p θ (y|x)] = 0! there are few ways that self-training can improve performance: recently in @xcite , it was shown that for machine translation, significant gains could be realized via self-training, and that most of this gain could be attributed to the last effect: in particular, invariance to input and dropout noise. in summary, predictive models can benefit substantially from self-training: one does not need to improve a generative model of the data to improve prediction performance on that data. nevertheless, we will next relate the predictive models used in qa generators and qa systems to underlying implicit generative models, to derive insight and justify existing qa filtering methods, and derive new ones. it turns out that several self-training algorithms for qa optimization actually optimize an approximate lower bound on the probability of the data, and so have the potential to evolve their representations to maximize the probability of the target data. ## a generative framework for adapting qa systems in this section we consider a simple generative framework for understanding predictive qa systems. in unsupervised data settings, only the context c is observed, and the questions {q} and answers {a} are latent, and must be inferred. here we consider the following generative model for each context: where the index of the context in the adaptation data has been omitted to avoid notational clutter. typically the factors of a generative models are explicit, and each factor is endowed with parameters, to maximize the probability of the data. in this paper, these factors, as we shall see, are only implicit, and we explore the possibility of maximizing the probability of the data, using only inference networks, which condition on the context: namely the answer generator, p(a|c), the question generator, p(q|a, c), and the qa system, p(a|q, c). our interest in doing so is 3 fold: 1) to avoid having to specify and train a generative model of contexts, which may be difficult to learn. 2) to better understand the theoretical underpinnings of existing augmentation algorithms such as the roundtrip consistency algorithm (rtc) @xcite , and 3) foremost, to achieve the end-goal of deriving insight that leads to more effective iterative algorithms for adapting qa systems. we begin our quest by low-bounding the probability of the data, with a posterior distribution q(a, q) over the latent questions and answers for the context, as is done for vaes @xcite . expanding the prior term p(q, a) in terms of question and answer generators, p(q|a, c) and p(a|c), we have: where p(c ) is the current estimate of probability of context c . in general we expect this lower bound to be quite strong, as p(q|a, c ) and p(a|c ) are identically or close to zero for most @xmath0. the resulting lower bound is on p(c) is: which can be maximized to improve p(c). the resulting bound looks much like a typical variational bound (e.g. 5), the principle difference being that the distributions being optimized are themselves inference networks, and condition on c. the effective joint distribution corresponding to this bound is: in contrast with how vaes are typically optimized, here we utilize multiple samples to form a posterior over qa pairs @xcite , and employ a form of prioritized truncated variational inference (tvi) to combat mode-dropping, which was recently shown to be a proper variational bound that can be iterated @xcite . tvi in short, utilizes a subset φ of joint states to define the posterior, i.e. q(q, a) ∝ p(q, a, c), {q, a} ∈ φ, 0 o.w, but any subset of states, and any convex set of weights, can be used to form a lower bound. when q(q, a|c) ∝ p(q, a, c), {q, a} ∈ φ, 0 o.w, the second bound is tight. in general q(q, a|c) takes the form: the consequence of this is that all the bounds derived above hold, even if the sums over all qa pairs are taken over a prioritized set. the process of defining q thus can be decomposed into two steps: ## training objectives typically when the end goal is data modeling, the probability of each data example is maximized indiscriminately. the maximum likelihood objective over the adaptation dataset {c t } in this case is: however, when training inference networks to make predictions for a given set of contexts in an unsupervised manner, it is natural to put more emphasis on confident predictions. a natural selftraining objective in this context is to weigh each context according to it's current estimated probability, so as to avoid re-enforcing poor predictions: with α ≥ 0. when α = 0, the standard ml objective is recovered. when α = 1, the normalization and the overall objective reduces to self-training on the current estimate of the joint distribution over {q, a, c} triples: we found that o st (1) consistently, leads to slightly better qa performance than pre-training with o m l , and so we optimize o st (1) during the pre-training step for all results in this paper, except where otherwise noted. as discussed above, summing over all qa pairs is intractable, and so we select the set φ by prioritizing wrt p(q|a, c)p(a|c) via beam search. given the set, we need to approximate p(q, a, c). here we approximate the likelihood function p(c|q, a) by the qa posterior p(a|q, c), which will attribute low scores to poor qa pairs. since the answers a are abstractive, this is a good approximation. with p(c|q, a) ≈ p(a|q, c), and φ prioritizing wrt p(q|a, c)p(a|c) via beam search, we utilize the tightest possible tvi bound on (7): q(q, a, c) ∝ p(a|q, c)p(q|a, c)p(a|c) (15) treating these prioritized samples as independent samples from p(q|a, c)p(a|c) 1 , the final weight on each sample after importance sampling is proportional to p(a|q, c). the resulting per example adaptation objective for our generators and qa system are given by: where p(a|q, c) = p(a|q,c) ( q,@xmath1,c)) α when the objective o st (α) is used (p(a|q, c) when α = 1). in essence, the generators are trained on the reward signal p(a|q, c). note that the expected gradient of the qa term is not zero, because the set set has been prioritized. in addition, dropout is used during training, to encourage invariance to dropout noise on generated inputs. adding noise to the input would similarly improve performance, but we leave this to future work. effectively this approach further prioritizes the set of questions produced by beam search based on the posterior distribution of the answer that the question was generated for, and encourages prediction consistency over noisy representations. ## interpreting existing approaches current techniques can be understood in the context of the above bounds, which provides new theoretical justification for these methods, and furthermore justifies the iterative extensions of these methods presented herein. ## related work several authors have recently investigated the use of synthetic data to improve question answering systems, with most of the work we are aware of either augmenting the training set, or generating synthetic qa pairs on auxiliary data @xcite @xcite @xcite @xcite . in most existing work, the generators are trained only on ground-truth data. one exception is @xcite , which jointly self-trains question given answer (rnn) and answer given question (attentive reader) models. another is @xcite , which introduces a domain classifier to adapt to target domain data. however, in contrast with our work, they train their qa system component only on ground-truth source data, because they found that synthetic data degrades qa system performance, whereas our pt qa systems trained only on synthetic data often outperform the baseline. in this work, we focus on using synthetic data to adapt to target data, and iteratively improve both the generators and the qa system during the adaptation process using pre-trained transformers, by alternating between self-supervised pre-training and ground-truth fine-tuning phases. the closest existing work to ours that we are aware of is the roundtrip consistency paper itself @xcite , whose "fine-tuning only" experiments parallel ours, in that they fine-tune pre-trained bert models to define the answer and question generators, and the qa system (squad2 gain 0.9 f1 with 3m qa pairs). note that, in contrast with our work, their generators were trained only on groundtruth data, and the augmentation process was not iterated. an important element of this work is to show that iterative roundtrip consistency filtering and the answer posterior filtering method proposed here approximately optimize a lower bound on the probability of the target domain data. in @xcite , the authors propose a generative model for question answering, which purposefully avoids the use of a qa decoder, p(a|q, c), instead inverting the generative model p(a|c), p(q|a, c) explicitly over a prioritized set of answers using beam search, so that an answer must be able to generate the question to be an answer. additional discriminative training is required to make the approach work well, but nevertheless, this desiderata is captured by both roundtrip consistency filtering and answer posterior filtering: yes, generated questions are verified by virtue of matching generated and predicted answers, but also answers are validated by the generators' ability to communicate it when the @xmath2. yet another manifestation of this idea is regularizing the question generator p(q|a, c) and question answering system p(a|q, c) to encourage them to be consistent during (supervised) training @xcite . in @xcite the authors investigate the use of unilm on squad and fine-tune unilm as a question generator to augment the training set with new questions for ground truth answers. our preliminary experiments indicate that this is a straightforward way to significantly boost baseline qa and q generator performance before adapting to target data, given that the ground truth answers are available, using the methods described herein. ## experiments 12.1 test scenarios to evaluate the efficacy of unsupervised selftraining for qa system adaptation, we consider the following scenarios: in this paper well matched conditions (wm-lt, wm-pt) are assessed by adapting squad-trained models on squad development data @xcite , and mismatched conditions by adapting squad-trained models on the portion of the natural-questions (nq) development data @xcite that contains abstractive short answers.