Evaluation of Question Generation Needs More References (2023)

Abstract: Question generation (QG) is the task of generating a valid and fluent question based on a given context and the target answer. According to various purposes, even given the same context, instructors can ask questions about different concepts, and even the same concept can be written in different ways. However, the evaluation for QG usually depends on single reference-based similarity metrics, such as n-gram-based metric or learned metric, which is not sufficient to fully evaluate the potential of QG methods. To this end, we propose to paraphrase the reference question for a more robust QG evaluation. Using large language models such as GPT-3, we created semantically and syntactically diverse questions, then adopt the simple aggregation of the popular evaluation metrics as the final scores. Through our experiments, we found that using multiple (pseudo) references is more effective for QG evaluation while showing a higher correlation with human evaluations than evaluation with a single reference.

## introduction question generation (qg) is the task of generating questions that are relevant to and answerable by given text. since qg can be applied in not only educational scenarios @xcite @xcite but also improving question-answering tasks @xcite @xcite , designing better qg frameworks and their automatic evaluations have gained more attention @xcite . however, previous qg works mostly evaluate their methods based on how similar the generated questions are to the gold reference questions @xcite @xcite , using n-gram-based similarity metrics, such as bleu @xcite and rouge @xcite . given a single reference, these metrics do not account for the lexical and semantic diversity of questions @xcite , showing poor correlation with human judgment @xcite @xcite . though prior works studied alternative metrics of leveraging language models, such as bertscore @xcite and bleurt @xcite , such metrics are limited in that the diversity of gold questions is only implicitly represented in the embedding space, rather than data space (or, raw questions). to explicitly compare with the diverse gold questions in the data space, we propose to augment the single reference question for evaluating qg frameworks, which we call multi-reference evaluation (mre), by leveraging the few-shot ability of large language models (llms) like gpt-3 @xcite and @xcite . though there have been efforts to augment references for improving evaluations, they are either limited in other text generation tasks, such as machine translation @xcite ) and question answering @xcite , or the methods are hard to be applied in question generation tasks, as naive llms generate some negative (toxic or erroneous) questions @xcite . therefore, we utilize llms for paraphrasing to augment a reference question, rather than generating new questions from the given context. to the best of our knowledge, we are the first to apply reference augmentation to evaluate the qg frameworks. we briefly summarize our main contributions as follows: ## conclusion & future work in this paper, we studied the problem of evaluating the question generation frameworks, and observed that automatically augmenting the reference question with large language models is surprisingly effective, showing higher correlations with human-annotated scores. though we evaluated the effectiveness of multiple reference evaluations for testtime evaluations, where the gold human score is given, we hope future research to explore other scenarios, such as measuring validation performance (asking how much the test performance can be actually improved) and multi-reference training as in @xcite . exploring other tasks (machine translation and document summarization) or generation methods (giving context and the reference question together to llms) would be interesting for future research. inapplicability for answer-unconditional qg frameworks: mre can't be applied to answerunconditional qg frameworks because it only augments the reference question by paraphrasing without considering other possible questions of supposing other answers. large computations: to generate multi-reference questions, our method requires inference of large language models, which results in huge computational costs. therefore, this can become burdensome as the test dataset grows.