Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation (2021)

Abstract: Question answering (QA) models for reading comprehension have achieved human-level accuracy on in-distribution test sets. However, they have been demonstrated to lack robustness to challenge sets, whose distribution is different from that of training sets. Existing data augmentation methods mitigate this problem by simply augmenting training sets with synthetic examples sampled from the same distribution as the challenge sets. However, these methods assume that the distribution of a challenge set is known a priori, making them less applicable to unseen challenge sets. In this study, we focus on question-answer pair generation (QAG) to mitigate this problem. While most existing QAG methods aim to improve the quality of synthetic examples, we conjecture that diversity-promoting QAG can mitigate the sparsity of training sets and lead to better robustness. We present a variational QAG model that generates multiple diverse QA pairs from a paragraph. Our experiments show that our method can improve the accuracy of 12 challenge sets, as well as the in-distribution accuracy.

## introduction machine reading comprehension has gained significant attention in the nlp community, whose goal is to devise systems that can answer questions about given documents @xcite @xcite . such systems usually use neural models, which require a substantial number of question-answer (qa) pairs for training. to reduce the considerable manual cost of dataset creation, there has been a resurgence of studies on automatic qa pair generation (qag), consisting of a pipeline of answer extraction (ae) and question generation (qg), to augment question answering (qa) datasets @xcite @xcite . for the downstream qa task, most existing studies have evaluated qag methods using a test set from the same distribution as a training set @xcite @xcite . however, when a qa model is evaluated only on an in-distribution test set, it is difficult to verify that the model is not exploiting unintended biases in a dataset @xcite . exploiting an unintended bias can degrade the robustness of a qa model, which is problematic in real-world applications. for example, recent studies have observed that a qa model does not generalize to other qa datasets @xcite @xcite . other studies have found a lack of robustness to challenge sets, such as paraphrased questions (gan and ng, 2019), questions with low lexical overlap @xcite , and questions that include noise @xcite . while existing studies have proposed data augmentation methods targeting a particular challenge set, they are only effective at the expense of the indistribution accuracy @xcite @xcite . these methods assume that the target distribution is given a priori. however, identifying the type of samples that a qa model cannot handle in advance is difficult in real-world applications. we conjecture that increasing the diversity of a training set with data augmentation, rather than augmenting qa pairs similar to the original training set, can improve the robustness of qa models. poor diversity in qa datasets has been shown to result in the poor robustness of qa models @xcite @xcite , supporting our hypothesis. to this end, we propose a variational qag model @xcite . we introduce two independent latent random variables into our model to learn the two one-to-many relationships in ae and qg by utilizing neural variational inference @xcite . incorporating the randomness of these two latent variables enables our model to generate diverse answers and questions separately. we also study the effect of controlling the kullback-leibler (kl) term in the variational lower bound for mitigating the posterior collapse issue @xcite , where the model ignores latent variables and generates outputs that are almost the same. we evaluate our approach on 12 challenge sets that are unseen during training to assess the improved robustness of the qa model. in summary, our contributions are three-fold: ## discussion and conclusion we presented a variational qag model, incorporating two independent latent random variables. we showed that an explicit kl control can enable our model to significantly improve the diversity of qa pairs. our synthetic datasets were shown to be noisy in terms of the grammaticality and answerability of questions, but effective in improving the qa performance in the in-distribution test set and the 12 challenge sets. while out synthetic datasets are noisy, they may unintentionally improve the robustness to the noise that can occur in real applications. however, we should pay attention to the negative effect of using our noisy dataset. for example, the lack of the answerability of our synthetic questions may lead to the poor performance in handling unanswerable questions, such as squad v2.0. moreover, qag methods led to improvements in most of the 12 challenge sets while being agnostic to the target distributions during training. we need to pursue such a target-unaware method to improve the robustness of qa models, because it is quite difficult for developers to know the types of questions a qa model cannot handle in advance. in summary, our experimental results showed that the diversity of qa datasets plays a nonnegligible role in improving its robustness, which can be boosted with qag. we will consider using unlabeled documents in other domains to further improve the robustness to other domain corpora in our future study.