Closed-book Question Generation via Contrastive Learning (2023)

Abstract: Question Generation (QG) is a fundamental NLP task for many downstream applications. Recent studies on open-book QG, where supportive answer-context pairs are provided to models, have achieved promising progress. However, generating natural questions under a more practical closed-book setting that lacks these supporting documents still remains a challenge. In this work, we propose a new QG model for this closed-book setting that is designed to better understand the semantics of long-form abstractive answers and store more information in its parameters through contrastive learning and an answer reconstruction module. Through experiments, we validate the proposed QG model on both public datasets and a new WikiCQA dataset. Empirical results show that the proposed QG model outperforms baselines in both automatic evaluation and human evaluation. In addition, we show how to leverage the proposed model to improve existing question-answering systems. These results further indicate the effectiveness of our QG model for enhancing closed-book question-answering tasks.

## introduction question generation (qg) has a wide range of applications, such as generating questions for exams @xcite @xcite or children's story books @xcite , recommending questions for users in a dialogue system @xcite , improving visual @xcite or textual question-answering tasks @xcite @xcite @xcite , asking clarification questions @xcite @xcite , and generating queries for sql @xcite or multimodal documents @xcite . ## related work many previous works on qg are under the openbook setting, which takes factoid short answers @xcite or human-generated short answers @xcite with the corresponding passages to generate questions @xcite . early approaches for question generation rely on rule-based methods @xcite . to bypass handcrafted rules and sophisticated pipelines in qg, @xcite introduce a vanilla rnn-based sequence-to-sequence approach with an attention mechanism. the recently proposed pre-trained transformer-based frameworks @xcite also improve the performance of qg. in addition, @xcite shows that the lexical and factual diversity of qg provides better qa training. however, their success can not directly adapt to the closed-book setting, where the model is supposed to generate questions solely relying on answers. in this work, we explore the widely applicable closed-book qg setting, which is still under-explored. contrastive learning aims to pull semantically similar neighbors close and push non-neighbors apart. it has achieved great success under both supervised and unsupervised settings. in pioneer works, the contrastive loss function @xcite has been proposed as a training objective in deep metric learning considering both similar and dissimilar pairs. recently, @xcite proposes the simclr framework to learn useful visual representations. viewing contrastive learning as dictionary look-up, @xcite present momentum contrast (moco) to build dynamic dictionaries for contrastive learning. some works apply contrastive learning into the nlp domain to learn better sentence representations @xcite . in addition, contrastive learning has been applied in multilingual neural machine translation @xcite , abstractive summarization @xcite , and multi-document question generation @xcite . the recent most relevant work is @xcite , where they design two contrastive losses for paraphrase generation. in this work, we adopt contrastive learning for improving representation learning in question generation under a closed-book setting. ## proposed approach to answer our research question -how can we empower a qg model to better understand the semantics of long-form abstractive answers and store more information in its parameters? -we propose a closed-book qg model, which generates questions directly without access to external knowledge. formally, given an answer sentence x, a closed-book qg engine generates a natural question y. figure 1 illustrates an overview of the proposed qg framework, which consists of three parts: question generation, contrastive learning, and answer reconstruction. the framework is optimized with the joint losses from these three parts simultaneously. ## experimental setup to evaluate the effectiveness of the proposed qg framework, we aim to answer the following research questions (rq) via experiments: rq1: can this proposed qg framework improve the performance of the closed-book qg tasks? rq2: are the generated questions of good quality? that is, are they fluent and relevant to the answer? rq3: can this qg framework be leveraged as a good resource to generate synthetic data for the qa task? rq4: how much does each component in the framework contribute? ## experiment results in this section, we answer the four experimental research questions in turn. ## case study finally, we showcase some examples of the generated questions on the wikicqa dataset. for the first example in ## conclusion in this work, aiming to empower a qg model to better understand the semantics of long-form abstractive answers and store more information in its parameters, we propose a closed-book qg model empowered by a contrastive learning module and an answer reconstruction module. we present a new closed-book long-form qa dataset -wikicqa involving more than 20k real user qa pairs and show that wikicqa is a valuable training resource, complementing public datasets. through the experiments, the proposed qg model shows better performance than baselines through automatic and human evaluations. moreover, we show how to leverage the proposed model as a data augmentation strategy to improve existing closed-book qa systems. the closed-book qa model, pre-trained on our generated synthetic qa pairs, achieves better performance on the seen dataset (wikicqa). in addition, it shows strong generalization on unseen datasets (gooaq and eli5), which further demonstrates the effectiveness of our qg framework for enhancing closed-book qa performance.