QGE val: Benchmarking Multi-dimensional Evaluation for Question Generation (2024)

Abstract: Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose **QGEval**, a multi-dimensional **Eval**uation benchmark for **Q**uestion **G**eneration, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.

## introduction question generation (qg) is a typical natural language generation (nlg) task that aims to generate natural language questions based on an input context and optionally an answer. qg has broad applications such as question answering (qa) @xcite , conversational systems @xcite , and knowledge assessment @xcite . however, it has been demonstrated that questions generated by qg models suffer from problems like ambiguities and hallucinations @xcite , passage: ...... the publication of a taoist text inscribed with the name of töregene khatun, ögedei's wife, is one of the first printed works sponsored by the mongols...... answer: töregene khatun reference: who was ögedei's wife? q1: who was the name of ögedei's wife? scores: flu. 1 : an example of qgeval, including a passage, an answer, a reference question, and 15 generated questions (only 3 are shown for brevity). the score ranges from 1 to 3 (higher better). errors within questions are highlighted with underlines. abbreviations are as follows. flu.:fluency; clar.:clarity; conc.:conciseness; rel.:relevance; cons.:consistency; ans.:answerability; ansc.:answer consistency. human evaluation is widely acknowledged as the gold standard for evaluating qg @xcite , with most automatic metrics striving to align their results with human evaluation results @xcite . however, the criteria of human evaluations are varied in existing research, leading to inconsistent and unreliable evaluations of qg models @xcite and automatic metrics @xcite . this inconsistency highlights the urgent need to establish a unified human evaluation benchmark to ensure reliable evaluations. despite the importance of such benchmarks, few have been published, and the existing ones usually have the following limitations: 1) focusing only on specific dimensions like answerability; 2) involving a small amount of data (e.g., <1k samples); 3) employing a limited variety of models to generate questions, resulting in a lack of diversity in the data. for instance, @xcite generated monotonous questions using rule-based methods for evaluation and focused primarily on the answerability of questions, neglecting other dimensions. gollapalli and ng (2022) evaluated generated questions from four dimensions but only included 500 questions generated by three models. @xcite utilized seven qg models to generate 1k+ questions to be evaluated, however, they merely assessed whether the generated questions could be accepted as reading comprehension quiz questions, rather than scoring them on multiple dimensions. to address the above issues, we propose qgeval, a multi-dimensional evaluation benchmark, which evaluates questions across 7 dimensions and contains 3k questions generated by 15 qg models (including llms) based on 200 passages and answers. specifically, through preliminary error analysis of the generated questions (described in section 2.2), we identified seven evaluation dimensions and categorized them into two aspects: 1) linguistic dimensions, including fluency, clarity, and conciseness, which are basic requirements that a natural language text should meet; and 2) task-oriented dimensions, including relevance, consistency, answerability, and answer consistency, which involve requirements specific to qg tasks. as illustrated in using qgeval to evaluate the performance of 15 different qg models, we find that these models perform relatively poorly in terms of answerability and answer consistency compared to other dimensions. we also evaluate and compare the performance of 15 existing automatic metrics, observing that there is still a gap between these metrics and human evaluations. to summarize, our main contribution is fourfold: ## the qgeval dataset in this section, we describe how we construct the qgeval dataset, the overall pipeline includes two stages: question generation and human evaluation, as shown in figure 1 . ## experiment and evaluation in this section, we conduct a series of analytical experiments and evaluations on qg models and automatic metrics with qgeval. we aim to address the following three research questions. ## related work automatic metrics automatic evaluation of qg is still dominated by reference-based metrics such as bleu @xcite and q-metric @xcite , which compute the similarity between generated questions and references. as qg is a one-to-many generation task, this type of metric can not evaluate questions that are different from the references @xcite . reference-free metrics like bartscore @xcite and qrelscore @xcite overcome this limitation, but they often assign a single overall score as the evaluation result, which is less interpretable and not comprehensive. unieval @xcite and gptscore @xcite are designed to evaluate generated texts from multiple interpretable dimensions, but they are not specifically designed for the qg task, and thus their performance in evaluating qg is limited. human evaluation in qg since existing automatic metrics are not effective enough to measure the quality of generated questions, human evaluation is frequently used in the field of qg @xcite . however, the human evaluation criteria provided by existing works are disparate, leading to inconsistent evaluation of generated questions. @xcite utilized answerability, fluency, and grammaticality to assess question quality, while ushio et al. (2022) employed grammatically, understandability, and answerability for evaluation. @xcite focused on consistency and diversity of generated questions. disparate human evaluation criteria also result in inconsistent evaluation and comparison of automatic metrics. @xcite proposed q-metric and computed the correlations between existing automatic metrics and human judgments on answerability. @xcite proposed qrelscore and compared it with other metrics based on their human evaluation results on three dimensions: grammaticality, relevance, and answerability. @xcite evaluated the performance of automatic metrics on their newly annotated data as the human evaluation of generated questions is not available in previous work. thus, it's urgent to develop unified and reliable human evaluation benchmarks to ensure consistent and accurate assessments of generated questions and automatic metrics. ## conclusion in this work, we introduced a comprehensive, multidimensional evaluation benchmark, qgeval, to facilitate the evaluation of generated questions from various models and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. it contains 3k questions generated from 15 different qg models. through analysis of qgeval, we found that most models performed unsatisfactorily on answerability and answer consistency. this highlights the importance of focusing on the two dimensions in future qg model designs. additionally, our evaluation of 15 existing automatic metrics revealed that these metrics still exhibit relatively low correlation coefficients with human annotation scores, emphasizing the need to explore advanced metrics that align better with human evaluation. we hope that this work will serve as a valuable resource for future research on question generation evaluation and models.