Reference-based Metrics Disprove Themselves in Question Generation (2024)

Abstract: Reference-based metrics such as BLEU and BERTScore are widely used to evaluate question generation (QG). In this study, on QG benchmarks such as SQuAD and HotpotQA, we find that using human-written references cannot guarantee the effectiveness of the reference-based metrics. Most QG benchmarks have only one reference; we replicate the annotation process and collect another reference. A good metric is expected to grade a human-validated question no worse than generated questions. However, the results of reference-based metrics on our newly collected reference disproved the metrics themselves. We propose a reference-free metric consisted of multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models. These criteria are not constrained to the syntactic or semantic of a single reference question, and the metric does not require a diverse set of references. Experiments reveal that our metric accurately distinguishes between high-quality questions and flawed ones, and achieves state-of-the-art alignment with human judgment.

## introduction question generation (qg) usually refers to the task of answer-aware question generation for controllability, aiming at generating a question based on a given context and answer span. solutions are used to improve educational tools, build a product-based question-answering (qa) database, etc. though anchored on a specific answer, there are still multiple ways of framing a question semantically and syntactically @xcite . users expect quality of every generated question. to evaluate qg performance, reference-based metrics are widely used, which assess a machinegenerated question against a human-written reference. the metrics are calculated either at the word level such as bleu @xcite , rouge @xcite , and meteor @xcite , or in the embedding space such as bertscore @xcite . the challenges of using these evaluation metrics speak to the metrics themselves, considering word overlaps and/or semantic similarity between the generated question and the reference. in this sense, a qg model can "cheat" on the metrics by using many similar words to the reference, but ignoring essential components of a question. @xcite the effectiveness of reference-based metrics, developed a qa model, and defined a new metric named "answerability" or rquge. though they showed a higher correlation with human preference, the failure of reference-based metrics was not studied, and the new metric's effectiveness is sensitive to the qa model's training and limited to its ability. to disprove existing metrics, the challenge can be traced to the lack of diverse references for benchmark datasets. previous works have shown that with access to a more diverse pool of references, the problem of poor correlation for these metrics can be mitigated @xcite @xcite . however, qg benchmarks often contain only one human-written ground-truth per example. our study starts from collecting another set of human-written references for two qg benchmarks, following their standard annotation instructions. besides the new references, we collect three groups of candidate questions, each lacking in an essential aspect of a question, for comparison. we study how five reference-based metrics, namely bleu-4 @xcite , bleurt @xcite , rouge-l @xcite , bertscore @xcite , and q-bleu @xcite , and two reference-free metrics, qascore @xcite , and rquge @xcite , score the four groups of questions. fig. 1 highlights the incompetency of current qg metrics in distinguishing the new ref-naco (ours) rquge qascore bertscore bleurt rouge-l q_bleu bleu metric 0.0 0.2 0.4 0.6 0.8 1.0 normalized metric value group 1: human-annotated question, different from refq which book about the early years of hillary rodham clinton was written by the founder of media matters for america? group 2: questions generated by gpt-3.5 that need 1-hop reasoning (lacking complexity) what is the name of the book about the early years of hillary rodham clinton written by david brock? group 3: non-questions generated by gpt-3.5 that use context words in refq (lacking naturalness) david brock founded media matters for america, a media watchdog group, and wrote a book in 1996. group 4: questions selected from training set randomly (lacking answerability) which state was the composer, who wrote the music and lyrics to "around the world," born? david @xcite ) is an american neo-liberal political operative, author, and commentator who founded the media watchdog group media matters for america. he has been described by "time" as "one of the most influential operatives in the democratic party". ## naco: a novel multi-dimensional reference-free qg metric based on extensive review of the human evaluation procedure in qg literature, detailed in appx. a.1, we identify three essential criteria of a question: naturalness, answerability, and complexity. we propose naco, which leverages prompting and chain-of-thought (cot) reasoning @xcite to obtain a score for each criterion. specifically, given the relevant context passage(s) and a question, we instruct an llm as follows: if a candidate question passes the initial naturalness and answerability evaluation, we determine whether its complexity aligns with expected standards for the domain and dataset. for example, in the hotpotqa dataset, questions that require multi-hop reasoning might be preferred over simpler, single-hop questions. this preference may not hold in other datasets. in this sense, naco relies on a subset of examples from the specific dataset to find the expected complexity of a question in that dataset. specifically, we perform the above cot-qa process to obtain the complexity of the references. expected complexity is then defined by the most common number of reasoning steps needed by the llm to answer a reference question. in our experiments, we use 750 examples from the training set of squad and hotpotqa to compute the expected complexity for each dataset. subsequently, naco measures the similarity, denoted by c cand , between the complexity of the candidate question and the expected complexity. overall, naco is a weighted combination of n cand , a cand , and c cand . in our experiments, we adopt a fair weight 1 3 for each criterion. we provide additional details on how c cand is computed and integrated into the final score in appx. a.3 4 experiments ## conclusion in this work, we questioned the competency of reference-based metrics in providing an accurate assessment for question generation. we replicated the data collection process used for benchmark datasets, gathering candidate questions qualified as new references. our analysis highlights the shortcomings of reference-based metrics in differentiating new references from flawed candidates, assigning significantly lower scores to the former. even the recently introduced reference-free metric, rquge, face difficulties in this regard. to address these challenges, we introduce naco, a multi-dimensional, reference-free metric bridging the gap between automated evaluation and human judgment in question generation. our experimental results showcase that naco, leveraging the chainof-thought capabilities of large language models for question answering, not only meets the expectations for quantitative qg metrics but also achieves state-of-the-art alignment with human evaluation.