Generative Language Models for Paragraph-Level Question Generation (2022)

Abstract: Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English.QG-Bench is released along with the fine-tuned models presented in the paper (https://github.com/asahi417/lm-question-generation), which are also available as a demo (https://autoqg.net/).

## introduction question generation @xcite is the task of generating a question given an input context consisting of a document, a paragraph or a sentence, and an answer where the question is anchored (see figure 1 ). qg has been widely studied in natural language processing communities @xcite @xcite , and it has recently been exploited to train question answering (qa) models without human supervision @xcite bansal, 2019; @xcite , or as a means of data augmentation @xcite . it has also been applied to develop educational systems @xcite , information retrieval models @xcite , and for model interpretation @xcite . despite its success in downstream applications, the development of neural qg models has received less attention. for example, the choice of the base pre-trained model is arbitrary (without proper justification in most cases) as it is not straightforward to compare different models. as a consequence, while ernie-gen @xcite and unilmv2 @xcite are current sota in the squad qg benchmark @xcite , t5 @xcite and bart @xcite are used in many applications in practice @xcite @xcite . a possible reason is inconsistent evaluation and comparison of qg models, due to the lack of appropriate evaluation protocols and benchmarks. for instance, evaluation of qg models relies on bleu4 @xcite , meteor @xcite , and rouge l @xcite , with human-made questions as references. however, some of these metrics may have low correlation with human judgements, especially when it comes to answerability, since they tend not to take the associated answer into account @xcite . moreover, qg applications can use different contexts as input, such as sentence-level @xcite vs paragraph-level @xcite ), or answer-aware @xcite vs answer-free @xcite . these are generally used interchangeably in the literature. to investigate how to tackle the issues previously raised, we introduce qg-bench, a collection of standard qa datasets unified into a single benchmark, including domain-specific datasets and for eight different languages ( § 3). we then use qg-bench to fine-tune various generative language models (lms) by formulating paragraph-level qg as a sequence-to-sequence generation task ( § 4), and measure their performance on in-domain and language-specific data ( § 5). finally, we present a multi-faceted analysis of our qg models by varying their input context size ( § 6.1), conducting a manual evaluation ( § 6.2), and studying their abilities for domain adaptation ( § 6.3). ## related work early work on qg was based on human-engineered templates @xcite and well-designed pipelines @xcite , but soon neural approaches took over by generating a question from a text in an end-to-end manner @xcite @xcite . the quality of qg models was later improved by masked lm pre-training @xcite where the encoder of the qg model is fine-tuned from pre-trained lms @xcite . recently, sequence-to-sequence lm pre-training has allowed to fully fine-tune qg models (both encoder and decoder), achieving sota performance @xcite @xcite . following the latest research in the literature, we focus on sequence-to-sequence lm-based qg models. qg can be applied to domain adaptation @xcite , knowledge-enhanced lm pre-training @xcite , adversarial/counterfactual data augmentation @xcite , and nearest neighbour qa systems @xcite . applications of qg go beyond qa, including semantic role labeling @xcite , visual qa @xcite , multi-hop question decomposition @xcite , and question rewriting @xcite . moreover, qg can be applied to unsupervised qa, which consists of training a qa model without any supervision and relying on a qg model to generate questions @xcite . @xcite showed that with a carefullydesigned qg model, we can generate high-quality qa datasets on which a qa model can even outperform their supervised counterparts. this inspired @xcite to propose qa-based evaluation, which connects the quality of a qg model to the accuracy of a qa model trained on the synthetic data generated by the qg model. while qg models can be applied to this variety of tasks, the comparison across tasks is not always straightforward. for this reason, and given the relevance of qg in current research, in this paper we propose an intrinsic qg benchmark in which we can evaluate different aspects of a qg model in a simple manner, including, but not only, analysis of input types, domain adaptability and multilinguality. the most similar work to ours is the mtg benchmark @xcite , which contains multilingual test sets for four nlg tasks. while qg is part of this benchmark, there are a few major differences from our proposed qg-bench: (i) we provide training/validation/test sets to allow model training in each language in addition to the evaluation; (ii) mtg's test set consists of parallel sentences across languages by a translation from english, while we leverage monolingual datasets; (iii) we include eight languages, while mtg has five; and (iv) qg-bench includes datasets from different domains and styles. ## lms for question generation in this section, we formalize the qg task from a language modelling perspective ( § 4.1), including details on the fine-tuning process ( § 4.2) and the setup for our experiments with qg-bench ( § 4.3). ## automatic evaluation in this section, we report the main results in qg-bench ( § 3), using the methodology described in § 4. ## analysis in this section, we complement the automatic evaluation with an extensive analysis on various relevant aspects of the question generation models. ## conclusion in this paper we presented qg-bench, a unified benchmark and evaluation for testing paragrahlevel qg models. the benchmark is composed of the general-purpose squad dataset, as well as domain-specific datasets of different styles for english. moreover, it includes language-specific datasets for eight different languages. using qg-bench as a reference, we tested recent generative language models on the task, and evaluated them across a range of automatic metrics. to complement the automatic evaluation, we performed a comprehensive manual evaluation to better understand the performance of models and the role of automatic metrics (e.g., our study shows there are better metrics than the popular bleu4 when it comes to qg). in general, our results show that lms have come a long way for qg, being very competitive (e.g., t5 attains an overall manual score of, respectively, 2.80, 2.93 and 2.95 in answerability, grammaticality and understandability on squad), but have room for improvement when dealing with different domains and styles, and especially on languages other than english. as future work, we will continue to study qg evaluation metrics in-depth to better understand what aspects we are missing when we use specific automatic metrics, using our manual evaluation as a proxy. moreover, the qg models analysed in this paper require an answer to be specified beforehand to generate the question. as a way to relax the constrain, we can train models for question and answer pair generation (qag) by generating the answer together with the question given a context. by generating both answers and questions together, new evaluation metrics would also be required to understand the validity and diversity of the answers selected, which we leave for future work.