Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAE s (2020)

Abstract: One of the most crucial challenges in question answering (QA) is the scarcity of labeled data, since it is costly to obtain question-answer (QA) pairs for a target text domain with human annotation. An alternative approach to tackle the problem is to use automatically generated QA pairs from either the problem context or from large amount of unstructured texts (e.g. Wikipedia). In this work, we propose a hierarchical conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts, while maximizing the mutual information between generated QA pairs to ensure their consistency. We validate our Information Maximizing Hierarchical Conditional Variational AutoEncoder (Info-HCVAE) on several benchmark datasets by evaluating the performance of the QA model (BERT-base) using only the generated QA pairs (QA-based evaluation) or by using both the generated and human-labeled pairs (semi-supervised learning) for training, against state-of-the-art baseline models. The results show that our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.

## introduction extractive question answering (qa) is one of the most fundamental and important tasks for natural language understanding. thanks to the increased complexity of deep neural networks and use of knowledge transfer from the language models pretrained on large-scale corpora @xcite @xcite , the stateof-the-art qa models have achieved human-level performance on several benchmark datasets @xcite . however, what is also paragraph (input) philadelphia has more murals than any other u.s. city, thanks in part to the 1984 creation of the department of recreation's mural arts program, . . . the program has funded more than 2,800 murals q1 which city has more murals than any other city? a1 philadelphia q2 why philadelphia has more murals? a2 the 1984 creation of the department of recreation's mural arts program q3 when did the department of recreation' s mural arts program start ? a3 1984 q4 how many murals funded the graffiti arts program by the department of recreation? a4 more than 2,800 question generation (qg), or question-answer pair generation (qag), is a popular approach to overcome this data scarcity challenge. some of the recent works resort to semi-supervised learning, by leveraging large amount of unlabeled text (e.g. wikipedia) to generate synthetic qa pairs with the help of qg systems @xcite @xcite . however, existing qg systems have overlooked an important point that generating qa pairs from a context consisting of unstructured texts, is essentially a one-to-many problem. sequence-tosequence models are known to generate generic sequences @xcite without much variety, as they are trained with maximum likelihood estimation. this is highly suboptimal for qag since the contexts given to the model often contain richer information that could be exploited to generate multiple qa pairs. to tackle the above issue, we propose a novel probabilistic deep generative model for qa pair generation. specifically, our model is a hierarchical conditional variational autoencoder (hcvae) with two separate latent spaces for question and answer conditioned on the context, where the answer latent space is additionally conditioned on the question latent space. during generation, this hierarchical conditional vae first generates an answer given a context, and then generates a question given both the answer and the context, by sampling from both latent spaces. this probabilistic approach allows the model to generate diverse qa pairs focusing on different parts of a context at each time. another crucial challenge of the qg task is to ensure the consistency between a question and its corresponding answer, since they should be semantically dependent on each other such that the question is answerable from the given answer and the context. in this paper, we tackle this consistency issue by maximizing the mutual information @xcite @xcite between the generated qa pairs. we empirically validate that the proposed mutual information maximization significantly improves the qa-pair consistency. combining both the hierarchical cvae and the infomax regularizer together, we propose a novel probabilistic generative qag model which we refer to as information maximizing hierarchical conditional variational autoencoder (info-hcvae). our info-hcvae generates diverse and consistent qa pairs even from a very short context (see but how should we quantitatively measure the quality of the generated qa pairs? popular evaluation metrics (e.g. bleu @xcite , rouge @xcite , meteor @xcite ) for text generation only tell how similar the generated qa pairs are to ground-truth (gt) qa pairs, and are not directly correlated with their actual quality @xcite . therefore, we use the qa-based evaluation (qae) metric proposed by @xcite , which measures how well the generated qa pairs match the distribution of gt qa pairs. yet, in a semi-supervised learning setting where we already have gt labels, we need novel qa pairs that are different from gt qa pairs for the additional qa pairs to be truly effective. thus, we propose a novel metric, reverse qae (r-qae), which is low if the generated qa pairs are novel and diverse. we experimentally validate our qag model on squad v1.1 @xcite , natural questions @xcite , and trivi-aqa @xcite datasets, with both qae and r-qae using bert-base @xcite as the qa model. our qag model obtains high qae and low r-qae, largely outperforming stateof-the-art baselines using a significantly smaller number of contexts. further experimental results for semi-supervised qa on the three datasets using the squad as the labeled dataset show that our model achieves significant improvements over the state-of-the-art baseline (+2.12 on squad, +5.67 on nq, and +1.18 on trivia qa in em). our contribution is threefold: ## related work question and question-answer pair generation early works on question generation (qg) mostly resort to rule-based approaches @xcite @xcite . however, recently, encoder-decoder based neural architectures @xcite have gained popularity as they outperform rule-based methods. some of them use paragraph-level information @xcite @xcite @xcite as additional information. reinforcement learning is a popular approach to train the neural qg models, where the reward is defined as the evaluation metrics @xcite , or the qa accuracy/likelihood @xcite @xcite . state-ofthe-art qg models @xcite @xcite use pre-trained language models. question-answer pair generation (qag) from contexts, which is our main target, is a relatively less explored topic tackled by only a few recent works @xcite @xcite . to the best of our knowledge, we are the first to propose a probabilistic generative model for end-to-end qag; yao et al. (2018) use vae for qg, but they do not tackle qag. moreover, we effectively resolve the qapair consistency issue by maximizing their mutual information with an infomax regularizer (belghazi et al., 2018; hjelm et al., 2019; yeh and chen, 2019), which is another contribution of our work. semi-supervised qa with qg with the help of qg models, it is possible to train the qa models in a semi-supervised learning manner to obtain improved performance. tang et al. (2017) apply dual learning to jointly train qa and qg on unlabeled dataset. yang et al. (2017) and tang et al. (2018) train qg and qa in a gan framework (goodfellow et al., 2014). sachan and xing (2018) propose a curriculum learning to supervise the qg model to gradually generate difficult questions for the qa model. dhingra et al. (2018) introduce a cloze-style qag method to pretrain a qa model. zhang and bansal (2019) propose to filter out low-quality synthetic questions by the answer likelihood. while we focus on the answerable setting in this paper, few recent works tackle the unanswerable settings. zhu et al. (2019) use neural networks to edit answerable questions into unanswerable ones, and perform semi-supervised qa. alberti et al. (2019) and dong et al. (2019) convert generated questions into unanswerable ones using heuristics, and filter or replace corresponding answers based on em or f1. variational autoencoders variational autoencoders (vaes) (kingma and welling, 2014) are probabilistic generative models used in a variety of natural language understanding tasks, including language modeling @xcite , dialogue generation @xcite @xcite @xcite , and machine translation @xcite @xcite . in this work, we propose a novel hierarchical conditional vae framework with an infomax regularization for generating a pair of samples with high consistency. ## method our goal is to generate diverse and consistent qa pairs to tackle the data scarcity challenge in the extractive qa task. formally, given a context c which contains m tokens, @xmath0, . . . , c m ), we want to generate qa pairs (x, y) where @xmath1, . . . , x n ) is the question containing n tokens and @xmath2, . . . , y l ) is its corresponding answer containing l tokens. we aim to tackle the qag task by learning the conditional joint distribution of the question and answer given the context, p(x, y|c), from which we can sample the qa pairs: we estimate p(x, y|c) with a probabilistic deep generative model, which we describe next. ## conclusion we proposed a novel probabilistic generative framework for generating diverse and consistent questionanswer (qa) pairs from given texts. specifically, our model learns the joint distribution of question and answer given context with a hierarchically conditional variational autoencoder, while enforcing consistency between generated qa pairs by maximizing their mutual information with a novel infomax regularizer. to our knowledge, ours is the first successful probabilistic qag model. we evaluated the qag performance of our model by the accuracy of the bert-base qa model trained using the generated questions on multiple datasets, on which it largely outperformed the state-of-theart qag baseline (+6.59-10.69 in em), even with a smaller number of qa pairs. we further validated our model for semi-supervised qa, where it improved the performance of the bert-base qa model on the squad by 2.12 in em, significantly outperforming the state-of-the-art model. as future work, we plan to extend our qag model to a meta-learning framework, for generalization over diverse datasets. the statistics and the data resource are summarized in squad we tokenize questions and contexts with wordpiece tokenizer from bert. to fairly compare our proposed methods with the existing semisupervised qa, we follow zhang and bansal (2019)'s split, which divides original development set from squad v1.1 (rajpurkar et al., 2016) into new validation set and test set. we adopt most of the codes from wolf et al. (2019) for preprocessing data, training, and evaluating the bert-base qa model. natural questions other than the original natural questions (kwiatkowski et al., 2019) dataset, we use subset of the dataset provided by mrqa shared task (fisch et al., 2019) for extractive qa. as semi-supervised setting with squad, we split the validation set provided from mrqa into half for validation set and the others for test set. all the tokens from question and context are tokenized with wordpiece tokenizer from bert. we generate qa pairs from context not containing html tag, and evaluate qa model with the official mrqa evaluation scripts. triviaqa for triviaqa (joshi et al., 2017), we also use the training set from mrqa shared task, and divide the development set from mrqa into half for validation set and the other for test set. all the tokens from question and context are tokenized with wordpiece tokenizer from bert. for evaluation, we follow the mrqa's official evaluation procedure. harvestingqa 3 we use paragraphs from harvest-ingqa dastaset (du and cardie, 2018) to generate qa pairs for qa-based evaluation (qae) and reverse qa-based evaluation (r-qae). for the baseline qg models such as maxout-qg and semantic-qg, we use the same answer spans from the dataset. for the experiments of maxout-qg baseline, we train the model and generate new questions from the context and answer, while the questions generated by semantic-qg are provided by the authors (zhang and bansal, 2019). c training details maxout-qg we use adam (kingma and ba, 2015) optimizer with the batch size of 64 and set the initial learning rate of 10 -3 . we always set the datasets train (#) valid (#) source squad 86,588 10,507 crowd-sourced questions from wikipedia paragraph natural questions 104,071 12,836 questions from actual userfor searching wikipedia paragraph triviaqa 74,160 7,785 question and answer pairs authored by trivia enthusaists from the web harvestqa 1,259,691 -generated by neural networks from top-ranking 10,000 wikipedia articles