End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems (2020)

Abstract: We propose an end-to-end approach for synthetic QA data generation. Our model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate both answers and questions. In a nutshell, we feed a passage to the encoder and ask the decoder to generate a question and an answer token-by-token. The likelihood produced in the generation process is used as a filtering score, which avoids the need for a separate filtering model. Our generator is trained by fine-tuning a pretrained LM using maximum likelihood estimation. The experimental results indicate significant improvements in the domain adaptation of QA models outperforming current state-of-the-art methods.

## introduction improving question answering (qa) systems through automatically generated synthetic data is a long standing research goal @xcite . although many past works have proposed different strategies for question generation, they have limited or no success in improving the downstream qa task @xcite @xcite @xcite @xcite . some recent approaches for synthetic qa data generation based on large pretrained language models (lm) have started to demonstrate success in improving the downstream reading comprehension (rc) task with automatically generated data @xcite . however, these approaches typically consist of multi-stage systems that use three modules: span/answer detector, question generator and question filtering. given an input passage, the span detector is responsible for extracting spans that will serve as answers for which questions will be generated. this module normally combines a pretrained qa model with handcrafted heuristics. the question generator is a large lm fine-tuned for the task of conditional generation of questions given passage and answer. the question filtering comprises another rc model that is used to score and filter the generated qa pairs. each module of this synthetic data generation pipeline is trained/tuned separately and errors from one stage can propagate to the next stages. additionally, each module is expensive to be computed because all use large transformer networks @xcite . in this work, we propose an end-to-end approach for synthetic qa data generation. our model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate both the answer and the question. in a nutshell, we feed a passage to the encoder and ask the decoder to generate the question and the answer token-bytoken. the likelihood produced in the generation process is used as a filtering score, which avoids the need of a separate filtering model. our generator is trained by fine-tuning a pretrained lm using maximum likelihood estimation (mle). we use bart @xcite as the pretrained lm in our experiments. we perform experiments with three different variations of our synthetic qa data generator: (1) aqgen, which generates first the answer then the question; (2) qagen, which generates first the question then the answer; (3) qagen two-step (2s), which generates first the question, concatenates it to the passage, then generates the answer in a second pass through the same encoder-decoder. we focus our empirical evaluation on the task of data augmentation for domain adaptation of reading comprehension (rc) models trained on squad 1.1 dataset. we assess the effectiveness of our qa data generators for domain adaptation of four different target domain datasets: natural questions (nq), bioasq, newsqa and duorc. we compare our results with recent work on domain adaptation for qa as well as with a three-stage synthetic data generator. qagen performs better than aqgen and the baselines for all datasets, while qagen2s provides the best results overall because it allows bidirectional attention between passage and question. for nq dataset, qagen2s improves the squad baseline by more than 8 points in em and more than 7 points in f1. for newsqa and bioasq the gains in em are also above 4 points. additionally, we also demonstrate that synthetically generated data by qagen2s can improve the in-domain performance of both small and large rc models, leading to f1/em improvements of 1/0.5 and 3.1/2.2 on roberta-large and bert-base-uncased trained rc models on squad dev. the main contributions of this work can be summarized as follows: (1) we propose the first effective end-to-end approach for synthetic qa data generation; (2) our approach solves an important issue in previous methods for qa data generation: the detection of good spans. we show that span detection can be effectively solved as a generation task, just like question generation; (3) as it uses a single end-to-end model, our data generation pipeline is simpler, faster and more efficient; (4) we perform comprehensive experiments that demonstrate the effectiveness of our proposed approach for domain adaptation of qa systems. ## end-to-end model for question and answer generation and filtering we model the problem of synthetic qa data generation as a conditional language modeling task. more specifically, we use an encoder-decoder (enc-dec) conditional lm as described in what follows. ## conclusions we presented a novel end-to-end approach to generate question-answer pairs by using a single transformer-based model. our experiments showed that by proper decoding, significant improvements in domain adaptation of rc models can be achieved. we concluded that using lm filtering improves the quality of synthetic question-answer pairs; however, there is still a gap with round-trip filtering with some of the target domains. improving lm-score-based filtering is a future direction of our work. while we were able to generate diverse, high quality and challenging synthetic samples on the target domains, the types of the questions produced still were limited to those of squad, since the generative models were trained on squad. it would be interesting to explore how one can adapt the generative models to the type of target domain questions.