An Empirical Comparison of LM -based Question and Answer Generation Methods (2023)

Abstract: Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context (e.g. a paragraph). This task has a variety of applications, such as data augmentation for question answering (QA) models, information retrieval and education. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an end-to-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches. However, there are differences depending on the underlying generative LM. Finally, our analysis shows that QA models fine-tuned solely on generated question-answer pairs can be competitive when compared to supervised QA models trained on human-labeled data.

## introduction question and answer generation (qag) is the task of generating a set of question-answer pairs given an input context such as a document, a paragraph or a sentence. qag can be applied to develop question answering (qa) models without human supervision @xcite @xcite and as a data augmentation mean for qa model understanding @xcite . moreover, qag is used as an aid of educational systems @xcite , to improve information retrieval models @xcite , and as a tool for model interpretation @xcite . qag stems from question generation (qg) @xcite @xcite , which consists of generating a question given an answer on the input context. despite qg being widely studied in the language model era @xcite figure 1 : overview of the considered qag approaches. @xcite , qag is a more complex task, since the answer needs to be generated and not assumed to be part of the input. therefore, it is unclear what types of qag models work in practice as no comprehensive comparisons have been established so far. in this paper, we formalize qag as a task that generates question-answer pairs given a context, and compare three simple qag strategies based on fine-tuning encoder-decoder language models (lms) such as t5 @xcite and bart @xcite . our three proposed approaches (illustrated in figure 1 ) consist of: (1) pipeline qag, which decomposes the task into answer extraction and question generation, learning a separate model for each subtask; (2) multitask qag, which uses a shared single model to train both subtasks instead of independent ones; and (3) end2end qag, which uses end-to-end sequenceto-sequence learning to generate question-answer pairs directly. finally, we compare these three approaches on a multi-domain qa-based evaluation, where qa models are trained with the questionanswer pairs that each qag model generates. all the qag models are publicly released via hug-gingface @xcite foot_0 , and available on the online demo foot_1 . ## related work there are a few works that leverage pre-trained lms for qag. for example, @xcite first fine-tuned bert @xcite on answer extraction and qg, and generate questionanswer pairs by extracting an answer, on which the associated question is generated. puri et al. (2020) followed a similar idea by fine-tuning an autoregressive lm for qg. in contrast, @xcite fine-tuned a single lm on answer extraction and qg jointly. @xcite trained an lstm sequence-to-sequence model from scratch to generate question and answer sequentially. more recently, @xcite used a qag model to generate adversarial examples for qa. similarly, @xcite improved on extractive qa by generating millions of question-answer pairs via qag. in these two last cases, the model to fine-tune was bart @xcite . while all these studies use the three methods that we analyse in this paper (i.e. pipeline, multitask and end2end), these are not easily comparable, as there are important differences among them in terms of settings, dataset, input to the lms, and evaluation metrics. moreover, except for @xcite , none of the proposed qag models have been made publicly available. finally, the two most recent studies using bart @xcite have not performed any evaluation on the qag model, as it is included as a part of a larger pipeline. we summarize the comparison of these prior works and our evaluation at ## question & answer pair generation given an input context c (e.g. a paragraph), qag aims to generate natural question-answer pairs q c related to the information in c: q @xmath0, a 1 ), (q 2 , a 2 ), . . . }. in what follows we describe three different approaches for qag based on fine-tuning language models. pipe. multi. e2e open eval. alberti et al. (2019 ) ✓ ✗ ✗ ✗ ✓ puri et al. (2020) ✓ ✗ ✗ ✗ ✓ lee et al. (2020) ✗ ✓ ✗ ✗ ✓ shakeri et al. (2020) ✓ ✗ ✓ ✗ ✓ bartolo et al. (2021) ✓ ✗ ✗ ✗ ✗ lewis et al. (2021) ✓ ✗ ✗ ✓ ✗ ours ✓ ✓ ✓ ✓ ✓ ## conclusion in this paper, we formalized qag as a task to generate pairs of questions and answers given an input context, and established baselines with three different qag approaches. to compare them, we conducted a multi-domain qa based evaluation that measures the performance of a qag model by fine-tuning qa models on the qa training dataset generated by the qag model. our evaluation shows that end2end qag models that generate questions and answers simultaneously are generally the most reliable. nonetheless, establishing a multitask paradigm with separation between answer extraction and question generation can have added benefits, especially when using lms such as bart. in general, the results are promising, as they show that these artificially-generated qa datasets rival in quality with those annotated by humans, which could save large amount of resources.