On the Importance of Diversity in Question Generation for QA (2020)

Abstract: Automatic question generation (QG) has shown promise as a source of synthetic training data for question answering (QA). In this paper we ask: Is textual diversity in QG beneficial for downstream QA? Using top-p nucleus sampling to derive samples from a transformer-based question generator, we show that diversity-promoting QG indeed provides better QA training than likelihood maximization approaches such as beam search. We also show that standard QG evaluation metrics such as BLEU, ROUGE and METEOR are inversely correlated with diversity, and propose a diversity-aware intrinsic measure of overall QG quality that correlates well with extrinsic evaluation on QA.

## question generation and diversity besides areas such as dialog @xcite and tutoring systems @xcite , automatic question generation (qg) has recently been applied with great success to generating synthetic training examples for question answering (qa) @xcite . yet an important question has remained unexplored: does increased textual diversity in automatically generated questions lead to better qa? in figure 1 we show four questions generated by one of our qg models (details in section 2) from a squad @xcite passage and an answer span (the qg prompt). the questions are different not only lexically, but also in what information about the answer entity they draw upon and even their use of world knowledge, e.g., tesla's reputation as a "mad scientist". intuitively, such sample diversity, if sufficiently accurate, could provide qa models with rich training signal. existing qg work has predominantly relied on customary beam search decoding for generation and n-gram similarity metrics such as bleu for evaluation @xcite @xcite on tesla' s 75th birthday in 1931, time magazine put him on its cover. the cover caption "all the world' s his power house" noted his contribution to electrical power generation. he received congratulatory letters from more than 70 pioneers in science and engineering, including albert einstein. in recent years, massively pre-trained neural language models (lms) @xcite @xcite have revolutionized nlp. in open-ended text generation, these models show remarkable robustness under sampling @xcite . this observation, coupled with the examples presented in figure 1 , suggests that treating qg for qa as a more open-ended generation problem and relying on the power of modern text generators to produce diverse yet accurate samples might yield better qa results than the current approach of optimizing for the "most likely" question. we test this hypothesis by fine-tuning a pretrained transformer-based masked lm @xcite for qg, and sampling questions from it using top-p nucleus sampling @xcite . other diversity-promoting text generation techniques exist-both at training time (e.g., vaes (kingma and welling, 2014)) and during inference (e.g., top-k sampling and diverse beam search @xcite )-that have been applied to various nlp tasks: language modeling @xcite , dialog @xcite , visual qg @xcite , image captioning @xcite and so on. we choose nucleus sampling because of its effectiveness, simplicity and speed. our experiments lead to the following discoveries: nucleus sampling indeed produces better qa results than beam search, even when only one question is generated per prompt. qg metrics that only reward similarity with gt are negatively correlated with diversity, and as a result, are inaccurate predictors of downstream qa performance of diversity-promoting qg. a measure of qg can be devised that combines diversity with similarity to gt, showing strong correlations with qa performance. ## question generation using roberta we fine-tune a roberta masked lm @xcite for qg given an answer span within a textual context (as shown in figure 1 ), and use nucleus sampling @xcite for generation. model: various transformer architectures can be used for text generation @xcite . following @xcite , we fine-tune a pre-trained masked lm as a prefix lm @xcite to predict a question token q t given (1) a prompt p 1:n : a tokenized textual context with special tokens delimiting an answer span, and (2) question tokens q 1:t 1 , if any, that have already been generated for the given prompt in a left-to-right order. a special separator token separates the question prefix from the prompt. the prompt is encoded using bidirectional attention and question tokens using causal (left-only) attention. we choose roberta as our pre-trained model because of its extended pre-training on large amounts of text @xcite . our implementation of the qg model is based on hugging face's @xcite pytorch implementation of roberta. fine-tuning: for each qg training example, the model is asked to predict a single question token q t given the prompt p 1:n , the previous question tokens q 1:t 1 (teacher-forced), and the mask m at timestep t. all questions end with an eos token that marks the end of generation. training attempts to minimize the masked lm loss, i.e., the negative log-likelihood of the gt token q t as the prediction for m in position t: inference: during generation, the fine-tuned roberta qg model outputs a probability distribution over the entire vocabulary at each question timestep t. top-p nucleus sampling (ns@p henceforth) samples from the (re-normalized) categorical distribution p n of the nucleus n, which is the smallest subset of vocabulary items that has (1) a cumulative probability mass greater than p, and (2) the highest probability among all such subsets: by restricting the pool to a high-likelihood region of the vocabulary, compared to top-k sampling, ns reduces the chances of generating low-probability items when the original distribution is peaked at one or a few items. our question generation works by repeated nucleus sampling of question tokens until @xmath0. ## experiments and results to test the effect of qg diversity on qa, we generate questions with both nucleus sampling and beam search from a number of different qg models and compare their performance. ## intrinsic evaluation of diverse qg to better understand the performance of existing generation metrics as measures of diverse qg, we take the set of all 32 samplers in figure 2 shows for all bins a rather poor, for some bins negative, median spearman's ⇢ score between the best qg metric (squad1-du: rouge-4, newsqa: rouge-1) and downstream qa f 1 . these results provide quantitative confirmation that rouge and similar metrics are inadequate evaluators of diverse qg for qa due to their sole focus on accuracy with respect to available gts. this leads us to our final research question: how to intrinsically measure the overall quality of qg for qa under diverse nucleus sampling? given the categorical distribution p n of vocabulary items in a model's nucleus n, we propose to measure both its accuracy (relative to gt) and diversity of generation. accuracy: similarly to lm perplexity, for timestep t of evaluation example s, we take the probability p n (q s,t | p, q s,1:t 1 ) of the model (more precisely, its nucleus n) generating the gt token q s,t , given prompt p and gt history q s,1:t 1 . we then average over all evaluation (s, t) pairs to compute model accuracy p (gt). diversity: an intuitive measure of the diversity of a model's nucleus n is the average entropy of p n over all evaluation timesteps. however, entropy is an unbounded measure, and has a non-linear inverse growth relative to our proposed accuracy metric, which makes their mathematical combination difficult. we instead rely on the observation that as we increase p in ns@p to make generation more diverse, the cardinality of n also goes up, on average, and so does the probability p (gt 2 n) that n contains the gt token. our experiments on both datasets showed that this measure of diversity, computed as the proportion of times n was found to include gt across all timesteps in the qg evaluation data, has high positive correlations with the entropy of p n (pearson's r: 98%-99%, spearman's ⇢: 87%-95%). note that unlike the accuracy metric p (gt), at each timestep t, the diversity metric p (gt 2 n) is boolean: the gt token is either in n or it is not. but importantly, its average across many evaluation timesteps is a probability measure of diversity, which enables a straightforward convex combination with our proposed accuracy metric. our final qg metric is a weighted sum of accuracy and diversity: w•p (gt)+(1 w)•p (gt 2 n), where w 2 [0, 1] is a tunable parameter reflecting the weight of accuracy relative to diversity. in our experiments, this metric outperforms all existing metrics by a large margin for a wide range of w values. in figure 2 , the median spearman's ⇢ score between this metric and qa f 1 in both in-domain (w=.7) and out-of-domain (w=.8) evaluation is over 90% for all bins. we observe similar performance differences between the proposed and existing metrics with pearson's r. given the scope of this paper, we evaluate the combined metric only on qg, but the underlying ideas apply to diverse text generation in general. further experiments are necessary to evaluate the metric on other generation tasks. ## conclusion while diversity of generation has received significant attention in other text generation problems (e.g., dialog), we show in this paper that it is also an important and measurable dimension of quality in question generation for qa. we hope that our work will encourage further exploration of diversity-promoting qg and its evaluation. possible future directions include a systematic study of different aspects of qg diversity (e.g., lexical and factual) and controlled diversification of individual aspects in generation.