Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval (2021)

Abstract: In this work, we introduce back-training, an alternative to self-training for unsupervised domain adaptation (UDA). While self-training generates synthetic training data where natural inputs are aligned with noisy outputs, back-training results in natural outputs aligned with noisy inputs. This significantly reduces the gap between target domain and synthetic data distribution, and reduces model overfitting to source domain. We run UDA experiments on question generation and passage retrieval from the Natural Questions domain to machine learning and biomedical domains. We find that back-training vastly outperforms self-training by a mean improvement of 7.8 BLEU-4 points on generation, and 17.6% top-20 retrieval accuracy across both domains. We further propose consistency filters to remove low-quality synthetic data before training. We also release a new domain-adaptation dataset - MLQuestions containing 35K unaligned questions, 50K unaligned passages, and 3K aligned question-passage pairs.

## introduction in domains such as education and medicine, collecting labeled data for tasks like question answering and generation requires domain experts, thereby making it expensive to build supervised models. transfer learning can circumvent this limitation by exploiting models trained on other domains where labeled data is readily available @xcite . however, using these pre-trained models directly without adapting to the target domain often leads to poor generalization due to distributional shift @xcite . to address this issue, these models are further trained on cheap synthetically generated labeled data by exploiting unlabeled data from target domain @xcite . one such popular data augmentation method for unsupervised domain adaptation (uda) is self-training @xcite . ## background in this section, we describe the source and target domain datasets, models for question generation and passage retrieval, and the evaluation metrics. ## transfer from source to target domain without adaptation we investigate how well models trained on natu-ralquestions transfer directly to our target domains without any domain adaptation. for comparison, we also present the results on naturalquestions. to be fair, we sample equal number of samples from the development set of naturalquestions as in the test set of mlquestions and pubmedqa for qg and ir tasks. figure 1 shows the results. we observe high performance drops across all generation metrics (14-20%) from naturalquestions (iid data) to mlquestions and pubmedqa (ood data). human evaluation on qg (see notation definition s, t source, target domain ps, pt source, target data distribution ds ≡ {(q i s , p i s )} m @xmath0 target unlabeled questions θ ≡ {θg, θr} qg, ir models sg, sr synthetic data for qg, ir ## unsupervised domain adaptation in this section, we describe self-training and backtraining methods to generate synthetic training data for unsupervised domain adaptation (uda). we also introduce consistency filters to further improve the quality of the synthetic data. ## domain adaptation evaluation as described in section 2, our source domain is nat-uralquestions and the target domains are mlquestions and pubmedqa. we evaluate if domain adaptation helps to improve the performance compared to no adaptation. we empirically investigate qualitative differences between self-training and backtraining to validate their effectiveness. we also investigate if consistency filters and iterative refinement result in further improvements. ## conclusion and future work we introduce back-training as an unsupervised domain adaptation method focusing on question generation and passage retrieval. our algorithm generates synthetic data pairing high-quality outputs with noisy inputs in contrast to self-training producing noisy outputs aligned with quality inputs. we find that back-training outperforms self-training by a large margin on our newly released dataset mlquestions and pubmedqa. one area of future research will be exploring back-training for other paired tasks like visual question generation @xcite and image retrieval @xcite , and style transfer @xcite from source to target domain and vice-versa. the theoretical foundations for the superior performance of back-training have to be explored further.