Exploring Question-Specific Rewards for Generating Deep Questions (2020)

Abstract: Recent question generation (QG) approaches often utilize the sequence-to-sequence framework (Seq2Seq) to optimize the log likelihood of ground-truth questions using teacher forcing. However, this training objective is inconsistent with actual question quality, which is often reflected by certain global properties such as whether the question can be answered by the document. As such, we directly optimize for QG-specific objectives via reinforcement learning to improve question quality. We design three different rewards that target to improve the fluency, relevance, and answerability of generated questions. We conduct both automatic and human evaluations in addition to thorough analysis to explore the effect of each QG-specific reward. We find that optimizing on question-specific rewards generally leads to better performance in automatic evaluation metrics. However, only the rewards that correlate well with human judgement (e.g., relevance) lead to real improvement in question quality. Optimizing for the others, especially answerability, introduces incorrect bias to the model, resulting in poorer question quality. The code is publicly available at https://github.com/YuxiXie/RL-for-Question-Generation .

## introduction question generation (qg) aims to endow machines with the ability to ask relevant and to-the-point questions about a document. qg has important practical applications, such as generating assessments for course materials in education @xcite , prompting user interaction in dialog systems @xcite , enabling machines to ask clarification questions such as faqs @xcite , and automatically building large-scale qa datasets for the research community @xcite . recent qg approaches @xcite @xcite have used seq2seq models with attention @xcite , which feeds the input document into an encoder, and generates a question about the document through a decoder. the training objective is to maximize the log likelihood of the ground-truth question paired with each input document using teacher forcing @xcite . however, as the ground-truth questions are insufficient to account for the many equivalent ways of asking a question, this likelihood-based training suffers from the problem of exposure bias @xcite , i.e., the model does not learn how to distribute probability mass over sequences that are valid but different from the ground truth. to address this issue, previous qg works proposed to optimize the model directly on question-specific rewards via reinforcement learning (rl). this process decouples the training procedure from the ground truth data, so that the space of possible questions can be better explored. moreover, it allows the training to target on specific properties we want the question to exhibit, such as relevant to a specific topic or answerable by the document. although various rewards have been employed for qg -such as bleu @xcite , the answerability reward @xcite , and the word movers distance @xcite ) -optimizing the reward scores does not always lead to higher question quality in practice, as observed by @xcite . how to define robust and effective qg-specific rewards still requires further investigation. we aim to analyze the effectiveness of question-specific rewards in qg. instead of using general natural language generation metrics such as bleu, we target three qg-related metrics that are commonly cited in human evaluations of question quality: (1) fluency indicates whether the question follows the grammar and accords with the correct logic; (2) relevance indicates whether the question is relevant to the document; and (3) answerability indicates whether the question is answerable given the document. we design a specific rl reward for each metric: a language model based reward for fluency, a discriminator-based reward for relevance, and a qa-based reward for answerability. after optimizing each reward via rl, we conduct comprehensive analysis, including automatic and human evaluation, to arrive at the following conclusions: (1) both individual and joint optimization of these rewards can lead to performance gain in automated metrics, but this does not guarantee an improvement in the real question quality; (2) the reward for relevance substantially helps to improve the question quality, while the reward for answerability reduces the quality due to the bias brought by the qa model; and (3) a reward is more likely to improve the question quality if the reward score correlates well with human judgement. ## related work early qg studies focused on using manually-designed rules or templates to transform a piece of given text to questions @xcite , with low generalizability and scalability. to address this, recent neural question generation (nqg) models take advantage of the seq2seq framework with attention, which are trained in an end-to-end manner, requiring far less labor and enabling better language flexibility. many improvements have been made to the original seq2seq nqg model @xcite , such as encoding answer information @xcite and incorporating linguistic features @xcite . a comprehensive survey of qg can be found in @xcite . among these attempts, utilizing rl to optimize qg-specific rewards has been adopted by recent works to address the exposure bias problem. to find a good proxy for question quality, various rewards have been proposed. one common type of reward is the similarity between the generated question and the reference question written by human. @xcite adopted bleu, rouge, and meteor as rewards. followup works employed more semantic-relevant metrics, such as the word movers distance @xcite and the paraphrasing probability @xcite . to generate more passage-relevant questions, @xcite designed a reward to measure the relevance between the input passage and the generated question based on their degree of overlapping. the answerability reward measures whether the generated question can be answered by the input passage. it is designed as either the confidence score that a pre-trained qa model can correctly answer the generated question @xcite , or the overlapping degree between the target answer and the answer predicted by the qa model @xcite . other types of rewards include @xcite , which train a discriminator to measure the naturalness, i.e., the question is human-written or generated. most question-specific rewards are empirically successful since they achieve performance gain in automatic evaluation metrics after rl training. however, this brings several followup questions that existing works have failed to answer: (1) does optimizing rl rewards really improve the question quality from the human standard, (2) which reward is more effective in improving the question quality, and (3) how the rewards interfere with each other when jointly optimized. this paper aims to bridge this gap through human evaluation and analytic experiments, aiming to provide a better understanding of how different rewards affect the question generation process. ## methodology given a document d as input, the objective is to generate a relevant question ŷ which can be answered by the document d. this is formulated as maximizing the conditional probability p(y|d): ## experiments we conduct experiments on hotpotqa @xcite , containing ∼100k crowd-sourced questions paired with wikipedia articles. generating a fluent, relevant, and answerable question in hotpotqa is a non-trivial task as it requires reasoning over different pieces of information in the input document. we follow the data split of @xcite to get 90,440 / 6,072 examples for training and testing, respectively. we further hold out 6,072 examples from the training data as the development set. the basic question generator is a seq2seq framework with copying @xcite , coverage @xcite , and attention @xcite mechanisms. we employ a 1-layer bi-directional gru as the encoder and a 1-layer gru as the decoder. we use the cased wordpiece tokenizer for the question generator following @xcite . the hidden size of the seq2seq model and the maximal input sequence length are set as 512 and 256, respectively. to train the language model used for evaluating the fluency reward, we fine-tune the pre-trained bert model @xcite on our target dataset, resulting in an lm with a perplexity of 8.85 on the dev set. our relevance discriminator, which is also fine-tuned from the pre-trained bert model, achieves a 91.16 f 1 score. the answerability discriminator based on spanbert-large obtains a 70.60 exact match (em) score and an 83.44 f1 score on the hotpotqa development set. in rl training, we empirically set the baseline rewards α f lu , α rel and α ans as -10, log(2), and log(2), respectively. when jointly training all the rewards, the trade-off parameters γ cov , γ f lu , γ rel and γ ans are tuned on the dev set and set to 0.25, 0.2, 1, 1, respectively. we provide ancillary material about supplementary experiments on hyper-parameter sensitivity and result analysis in our codebase foot_0 . other settings for training follow the standard best practice foot_1 . ## conclusion in this paper, we optimize three question-specific rewards via reinforcement learning on a seq2seq based question generator, aiming to improve the fluency, relevance and answerability of the generated questions. through comprehensive analytic experiments, including automatic and human evaluation, consistency validation, and meso analysis, we show that the effectiveness of a reward is poorly reflected by automatic evaluation metrics such as bleu. instead, we find a reward that correlates well with the human judgement generally has better effects on improving the question quality. in future works, we believe these observations can help to guide the design of other qg-specific rewards that target on unexplored aspects of question generation, such as the informativeness and the utility of questions.