Diversify Question Generation with Retrieval-Augmented Style Transfer (2023)

Abstract: Given a textual passage and an answer, humans are able to ask questions with various expressions, but this ability is still challenging for most question generation (QG) systems. Existing solutions mainly focus on the internal knowledge within the given passage or the semantic word space for diverse content planning. These methods, however, have not considered the potential of external knowledge for expression diversity. To bridge this gap, we propose RAST, a framework for Retrieval-Augmented Style Transfer, where the objective is to utilize the style of diverse templates for question generation. For training RAST, we develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward. Here, the consistency reward is computed by a Question-Answering (QA) model, whereas the diversity reward measures how much the final output mimics the retrieved template. Experimental results show that our method outperforms previous diversity-driven baselines on diversity while being comparable in terms of consistency scores. Our code is available at https://github.com/gouqi666/RAST .

## introduction question generation (qg) aims to generate questions from a given answer and a grounding paragraph. as a dual task of question answering (qa), qg can potentially be used for the automatic construction of qa datasets, thereby improving qa with little annotation effort @xcite @xcite . furthermore, qg can be utilized for educational purposes @xcite , dialog systems @xcite , and conversational recommendation systems @xcite . qg systems are typically known to suffer from two major issues, namely inconsistency and lack retrieve 3: what's the [mask] of [mask]'s the grecian saloon? generate 3:what's the year of tugh temur's death? retrieve 1: kvarner when was [mask]'s [mask]? generate 1:when was tugh temur's death? retrieve 2: in what [mask] did [mask] let their [mask] with [mask]? generate 2: in what year did tugh temur die? vanilla generate: when did tugh temur die? extractive template: when did [mask] die? retrieve 4: the [mask] to [mask] on the [mask] passed away in which [mask]? generate 4:the successor to kublai khan on the throne passed away in which year? of diversity. the former indicates that qg systems may yield context-irrelevant or answer-irrelevant questions @xcite @xcite . the latter is because qg systems may fail to capture the one-to-many nature of qg tasks; that is, many questions can be asked given the same pair of context and answer. existing solutions mainly exploit the internal knowledge within the context @xcite , the language model @xcite , or the semantic word space @xcite for diverse content planning. unfortunately, since these methods rely on obscure factors such as the black-box language model or the latent variable, they are not as controllable as exploiting external question templates (figure 1 ). in this paper, we aim to improve generation diversity by looking for expression variations in an external set of question templates. figure 1 shows several questions that can be generated with a number of retrieved templates for a given source context. although external information has been exploited for qg @xcite , prior methods depend on manually crafted set of type-dependent templates @xcite or paraphrasing samples @xcite . in contrast, we neither require the annotation of question types and the relevant templates (unlike @xcite ) nor assume that question rewriting samples are accessible (unlike @xcite ). our framework contains three main components: (1) a vanilla generator for initial template planning; all in all, our main contributions are three-fold: retrieval-augmented generation there has been a growing interest in integrating (external) knowledge from a retrieval model into a parametric language model for text generation. @xcite propose a retrieve-then-edit paradigm, where an editor is trained to modify a retrieval result to produce a more consistent response. @xcite exploit skeletons to diversify text generation outputs, where a skeleton is obtained by masking query-specific information in the text. the retrieval-augmented generation approach has also been used for task-oriented dialogs @xcite @xcite @xcite . these studies, however, either exploit surface matching methods (e.g. tf.idf) @xcite @xcite or separately train the retrieval with relevant labels @xcite @xcite . the retriever, therefore, might not be optimal for generation. several studies have jointly trained the retriever and the generation model @xcite @xcite , but they have mostly focused on consistency, not diversity. reinforcement learning for generation reinforcement learning (rl) has been used for text generation to mitigate the exposure bias issue associated with the standard supervised learning (sl) approach. here, the exposure bias refers to the fact that generation during inference relies on predicted tokens instead of ground-truth tokens as in training. furthermore, instead of optimizing proxy losses as in sl approach, rl directly optimizes the quality of interest via rl rewards, thus bridging the evaluation gap between training and testing. researchers have proposed various rl rewards for qg, including answerability (for question generation) @xcite , bleu-4 and word mover distance (wmd) @xcite , naturalness @xcite , consistency using a question paraphrase probability (qpp), and a question answering probability (qap) @xcite @xcite . previous methods have primarily focused on evaluating the consistency of generated questions, while we seek to evaluate both consistency and diversity. here, the diversity is achieved by training a retrieval model for a retrieval-augmented generation. our work is closely related to retgen @xcite . this method, however, differs from ours in several ways: 1) only the retrieval model is optimized using rl in retgen, whereas both the retrieval and the generation model are updated end-to-end via our rl framework; 2) it uses the likelihood of ground truth generation outputs as returns to update the retriever while we combine consistency and diversity rewards. text style transfer our objective of question style transfer bears some resemblance to text style transfer studies @xcite @xcite . the main difference is that we do not have predefined style labels, whereas most studies in the style transfer literature rely on given labels such as positive/negative or formal/informal. paraphrase generation paraphrasing involves transforming a natural language sentence into a new sentence with the same semantic meaning but a different syntactic or lexical surface form. although diversity can be obtained by paraphrasing generated questions, our setting is different from @xcite @xcite . specifically, question paraphrase datasets, such as @xcite , do not associate context with each pair of (sentence, paraphrased sample). as such, paraphrasing in these datasets can only focus on different word choice or syntactic modification of an input question. in contrast, our consistency reward allows generating questions as long as the answer is the same with the input question given the context. in other words, our method also pays attention to different clues of the context for qg diversity. ## two stage training we train rast using rl to avoid the exposure bias and the evaluation discrepancy between training and testing, which are often associated with supervised learning objectives @xcite . to accelerate the convergence of rl-based training, we first use supervised learning to initialize the style transfer model, resulting in a two-stage training procedure described in the following. ## conclusion this paper proposes rast, a framework that exploits question templates from an external corpus to improve expression diversity in qg systems. compared to previous methods, that exploit internal knowledge of language models for diversity, rast provides a more flexible and interpretative way to control the generation outputs. to train rast without question rewriting samples, we develop a novel rl based method, where we directly optimize the combination of the consistency and diversity rewards. in addition, we provide two stage training and diversity-driven sampling, which help better train our rl-based model. experiment results show that rast outperforms strong baselines in terms of diversity whereas being comparable on consistency scores. for future studies, we aim at further improvement by developing efficient training with a small number of paraphasing samples.