RQUGE : Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023)

Abstract: Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing a high score when there is a considerable lexical overlap or semantic similarity between the candidate and the reference questions. This approach has two major shortcomings. First, we need expensive human-provided reference questions. Second, it penalises valid questions that may not have high lexical or semantic similarity to the reference questions. In this paper, we propose a new metric, RQUGE, based on the answerability of the candidate question given the context. The metric consists of a question-answering and a span scorer modules, using pre-trained models from existing literature, thus it can be used without any further training. We demonstrate that RQUGE has a higher correlation with human judgment without relying on the reference question. Additionally, RQUGE is shown to be more robust to several adversarial corruptions. Furthermore, we illustrate that we can significantly improve the performance of QA models on out-of-domain datasets by fine-tuning on synthetic data generated by a question generation model and reranked by RQUGE.

## introduction given the context (e.g. paragraph), the goal of question generation (qg) is to generate questions with or without providing the answer spans. automatic question generation can be used in several applications: improving the question answering (qa) task @xcite @xcite , automatic assessments @xcite , especially for the educational domain @xcite , and the evaluation of factual consistency in the text generation tasks @xcite @xcite . previous work @xcite @xcite has shown that qg models can gen-erate questions inconsistent with the corresponding context and the answer span. so, measuring the acceptability of candidate questions is a critical challenge. human judgment is the most accurate method in natural language generation, but it is expensive, time-consuming, and not scalable. consequently, several metrics e.g. bleu @xcite , rouge @xcite , bertscore @xcite are proposed to automatically measure the quality of the generated text. specifically for the question generation task, previous work has utilised reference-based metrics e.g. bleu, rouge, bertscore, and bleurt @xcite @xcite to evaluate the quality of the candidate question given the reference question. however, these methods highly depend on the diversity of the reference questions for a given answer span. due to the huge cost of human annotations, existing qa/qg datasets mostly provide one reference question for the given context and answer, which results in wrongly penalising some valid questions. in figure 1 , the first candidate question (q 1 ) is generated by paying attention to different evidence in the context, and q 2 is a paraphrase of the reference, but both bleu and bertscore fail to assign high scores to them. furthermore, reference-based metrics are not sensitive to very small corruptions of the reference questions, which makes the candidate question unacceptable (q 3 ). in this paper, we propose rquge , a referencefree question generation evaluation metric that can compute the quality of the candidate question without requiring a reference question. given the corresponding context and answer span, our metric calculates the acceptability score by applying a general question-answering module, followed by a span scorer. the former module generates the answer span for the given candidate question, and the latter computes the semantic similarity of the predicted and gold answer spans. our metric is extremely valuable in cases where the reference question is not well-formed foot_0 or there is one (or no) reference for a given context and answer span. we evaluate our metric on several datasets, including squad (v1) @xcite , natural questions (nq) @xcite , and ms- @xcite , and show that it consistently has a better correlation with human judgment compared to previous qg metrics. we also integrate rquge into the decoding step by re-ranking candidate questions of each instance by our metric, leading to a better correlation with the human evaluation. additionally, we demonstrate that rquge is more robust to adversaries than previous metrics with +13.1% relative improvement. finally, we improve the performance of question answering models on an out-of-domain dataset by fine-tuning them on synthetic data generated by a question generation model, then re-ranked with rquge to choose the best candidate question for the given answer span, resulting in an +18.3% f1 and +22.2% em relative improvement. to sum up, our contributions are as follows: ## related work previous work on automatic evaluation of natural language generation (nlg) tasks have been categorized as follows: unsupervised metrics. it contains the most commonly used metrics e.g. bleu @xcite , rouge @xcite , chrf @xcite , and meteor (denkowski and lavie, 2010). these metrics calculate the correlation of reference and predicted sequences in a discrete space by utilising token-level matching functions. then, recent work e.g. bertscore @xcite and moverscore @xcite use bert @xcite embeddings to provide a soft token-level matching instead of the hard n-gram overlap. these metrics have been applied to various nlg tasks @xcite @xcite @xcite @xcite . specifically for qg evaluation, @xcite propose a scoring function, focusing on the answerability of the candidate question, which improves the human correlation when integrated with existing unsupervised metrics. regression-based metrics. these metrics e.g. comet @xcite , bleurt @xcite , s3 @xcite , and vrm @xcite train a regression layer in a supervised manner to mimic the human judgment. ranking-based metrics. the aim of these metrics is to assign a higher score to a better candidate compared to worse predictions. the most popular ones include beer @xcite and comet @xcite . generation-based metrics. the idea is to formulate the evaluation of nlg, as a text generation problem from pre-trained language models. given the source sequence, the better candidate should be generated with a higher score (probabil-ity) compared to the worse ones. the most popular ones are bartscore @xcite and prism @xcite . additionally, we include ctc (deng et al., 2021) and qrelscore @xcite as referencefree metrics for better comparison. ctc @xcite proposes an evaluation framework for nlg tasks by providing several reference-free metrics, which are computed by aggregating the alignment scores between the input, context and the predicted sequence. to measure the alignment, ctc @xcite uses bert @xcite embedding matching, discriminative, and regression models. 3 qrelscore computes the answerability of the candidate question by applying word-level hierarchical matching and sentence-level prompt-based generation. different from previous work, rquge combines question answering and span scoring modules to compute the acceptability of the candidate, which leads to a significantly better correlation with human judgement in multiple datasets with different domains and answer lengths. ## rquge architecture rquge architecture is illustrated in figure 2 . it consists of two components: question answering and span scorer modules. given the context, gold answer span, and the candidate question, generated by a question generation model (qg), rquge computes the acceptance score (κ) of the candidate question as follows: ## experimental setup datasets. we evaluate metrics on three widelyused qa datasets, including squad(v1) @xcite , nq @xcite , and ms- @xcite . nq is used to demonstrate the benefit of our metric in cases where reference questions are not wellformed and are derived from the google engine. 6 for ms-marco, we use description type of the dataset to show the effectiveness of our metric on candidate questions with long answer spans (13 tokens on average). unlike squad and nq, ms-marco is not included in the training data of the question answering module of rquge (i.e. unifiedqav2 @xcite ). we use ms-marco to demonstrate that rquge can be generalised to out-of-domain datasets. 7 question generators. we fine-tune two commonly used qg models, including gpt2 @xcite , trained with causal language modelling objective, and mixqg @xcite , which is the state-of-the-art question generator and is a t5-based @xcite sequenceto-sequence model. we choose gpt2 and mixqg as our question generators as there is a significant gap in their performance, making them suitable for evaluating the metrics. 8 baselines. we include bleu-4 @xcite , rouge-1, rouge-l @xcite , me-teor (denkowski and lavie, 2010), mover-score @xcite , and bertscore @xcite foot_7 as unsupervised metrics, that are commonly used for the question generation task. we additionally use qbleu, which is specific for information about questions that can be answered by each text span. the pre-trained model is available at https: //github.com/salesforce/qafacteval . qg evaluation. furthermore, we utilise bleurt-20 @xcite , and comet @xcite as regression-based and ranking-based metrics. finally, we include bartscore @xcite , fine-tuned on parabank2 @xcite . in all aforementioned metrics, scores are calculated between the candidate and reference questions. as reference-free baselines, we use qrelscore @xcite , and also adopt the factual consistency scorer of ctc @xcite to calculate the consistency score as: inspired by previous work @xcite , we ask annotators to score each candidate question based on three criteria: grammaticality, answerability, and relevance. grammaticality measures the syntactic structure of the question. answerability checks whether the question contains all the important entities, and relevance checks the relatedness of the generated questions with the given answer span. grammaticality and answerability scores are on a 3-point scale (3 as acceptable, and 1 as rejection), and relevance is on a 2-point scale. we sample 600 questions generated from fine-tuned qg models on squad(v1) @xcite , nq @xcite , and ms- @xcite datasets. we then randomly shuffle and anonymise them for annotators. further details of the human annotation procedure are provided in appendix c. 12 ## results and discussion we evaluate our rquge and previous metrics on various datasets and tasks. first, we evaluate the correlation of metrics with human judgment in sections 5.1 and 5.2. we then demonstrate their robustness on the adversarial subset in section 5.3. finally, section 5.4 illustrates that fine-tuning qa models on the synthetic data, created by our metric, improves their performance on out-of-domain datasets. ## conclusion we propose rquge , reference-free question generation evaluation metric to measure the quality of the generated questions, by better encoding the relevant context and answer without requiring a reference question. it consists of two modules, a question answering model, and a span scorer, which are existing pre-trained models without further fine-tuning. we compare the performance of rquge with existing qg metrics on squad, ms-marco, and nq datasets, and show that rquge achieves a significantly better correlation with human judgment. additionally, we integrate rquge into the decoding step by using it to re-rank the candidate questions, which leads to a better correlation with human. for robustness, we evaluate qg metrics on adversarial data by corrupting the reference questions and show that rquge achieves significantly better performance compared to previous work. finally, we show that fine-tuning qa models on the synthetic data, generated with a qg model and re-ranked with rquge , improves the performance of qa models on out-of-domain datasets.